THE afec-PROBLEM FOR GABOR SYSTEMS 

XIN-RONG DAI AND QIYU SUN 

Abstract. A Gabor system generated by a window function 
/y-\ and a rectangular lattice aTL x TLjh is given by 

^ ^(0, aZ X Z/6) := {e"^""*/V(i - "^a) : (m, 7i) e Z x Z}. 

CN One of fundamental problems in Gabor analysis is to identify win- 

\^ dow functions and time-frequency shift lattices al x Ijh such 

Mh that the corresponding Gabor system t/(0, oL x Z/6) is a Gabor 

■^ frame for L^(M), the space of all square-integrable functions on 

Qs the real line M. The range of density parameters a and h such 

^Sl that the Gabor system Q{(t),aZ x Z/6) is a frame for L^(IR) is 

fully known surprisingly only for few window functions, including 

r , the Gaussian window and totally positive windows. Janssen's tie 

^ suggests that the range of density parameters could be arbitrarily 

^ complicated for window functions, especially outside Feichtinger 

O algebra. An eye-catching example of such a window function is the 

ideal window function on an interval. In this paper, we provide 

^—1 a full classification of triples (a, b, c) for which the Gabor system 

K*" G{xi: o-'Z X I'/b) generated by the ideal window function xi on an 

^^ interval / of length c is a Gabor frame for L^(K). For the clas- 

sification of such triples {a,b,c) (i.e., the afoc- problem for Gabor 
systems), we introduce maximal invariant sets of some piecewise 
linear transformations and establish the equivalence between Ga- 
bor frame property and triviality of maximal invariant sets. We 

^^ then study dynamic system associated with the piecewise linear 

transformations and explore various properties of their maximal 
invariant sets. By performing holes-removal surgery for maximal 
invariant sets to shrink and augmentation operation for a line with 

/\ marks to expand, we finally parameterize those triples (a, b, c) for 

which maximal invariant sets are trivial. The novel techniques in- 
volving non-ergodicity of dynamical systems associated with some 
novel non-contractive and non-measure-preserving transformations 
lead to our arduous answer to the a6c-problem for Gabor systems. 
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1. Introduction 

Let L^ := L^(]R) be the space of all square-integrable functions on 
the real line M with the inner product and norm on L^ denoted by (■, ■) 
and II ■ II 2 respectively. In this paper, a Gabor system generated by a 
window function G L^ and a rectangular lattice aZ x Z/6 is given by 

(1.1) g{(j), oL X Z/6) := {e~2™*/V(^ - ^«) : (m, n) G Z x Z} 

([H HSl [TT] [20]); a frame for L^ is a collection T of functions in L^ 
satisfying 

/ -r^ \ 1/2 /,r^ \ 1/2 
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([HI E, El [13]); and a Gahor frame is a Gabor system that forms a 
frame for L^ ( [2TI [2^ ) . Here we use 1 /h to label the frequency spacing 
instead of h in other sources on Gabor theory, due to the convenience 
to state our full classification of triples (a, 6, c) for which the Gabor 
system Qixh oZ, x Z/6) generated by the ideal window function xi oii 
an interval / of length c is a Gabor frame for L^. 

Gabor frames have links to operator algebra and complex analysis, 
and they have been shown to be suitable for lots of applications involv- 
ing time-dependent frequency. The history of the Gabor theory could 
date back to the completeness claim in 1932 by von Neumann [301 P- 
406] and the expansion conjecture in 1946 by Gabor [T^l Eq. 1.29], 
with both confirmed to be correct in later 1970. Gabor frames be- 
come widely studied after the landmark paper in 1986 by Daubechies, 
Grossmann and Meyer, where they proved that for any positive density 
parameters a, h satisfying a/h < 1 there exists a compactly supported 
smooth function such that Q{4>, oL x Z/6) is a Gabor frame |12j . 

One of fundamental problems in Gabor analysis is to identify window 
functions and time-frequency shift sets such that the corresponding 
Gabor system is a Gabor frame. Given a window function G L^ and 
a rectangular lattice oL x Z/6, a well-known necessary condition for the 
Gabor system ^(0, oL x Z/6) to be a Gabor frame, obtained via Banach 
algebra technique, is that the density parameters a and h satisfy a/h < 1 
[21 El 1231 ESI EI] ■ But that necessary condition on density parameters 
is far from providing an answer to the above fundamental problem. 

The range of density parameters a, b such that the Gabor system 
Q{(f),a'L X Tj/h) is a frame for L^ is fully known stunningly only for 
small number of window functions (p jIEl [26l [271 ESI [33l [3l] . Among 
those, the Gaussian window v^exp(— vrt^) has received special atten- 
tion [151130]. It is conjectured by Daubechies and Grossmann ^\A\ and 
later proved independently by Lyubarskii [29j and by Seip and Wall- 
sten [33l [3l] via complex analysis technique that the Gabor system 
^(v2exp(— vrt^), aZ x Z/6) associated with the Gaussian window is a 
Gabor frame if and only if a/h < 1. A significant advance on the range 
of density parameters was recently made by Grochenig and Stockier 
that for a totally positive function of finite type, ^(0, aZ x Z/6) is a 
Gabor frame if and only if a/h < 1 [18] . 

For a window function in Feichtinger's algebra, an important re- 
sult proved by Feichtinger and Kaiblinger [13] states that the range of 
density parameters a, h such that the Gabor system Q{4>, aZ x Z/6) is a 
frame for L^ is an open domain on the plane. But for window functions 
outside the Feichtinger algebra, the range of density parameters could 
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be arbitrarily complicated, c.f. the famous Janssen's tie [25]. A strik- 
ing example of such a window function is the ideal window function on 
an interval [6l[l9l[25]. 

Denote by xe the characteristic function on a set E. Recall that 
given an interval /, ^(x/,ctZ x Z/6) is a Gabor frame if and only if 
Q{Xi+di(^'^ X Z/&) is a Gabor frame for every d G M. By the above 
shift-invariance, the interval I can be assumed to be half-open and 
have zero as its left endpoint, i.e., I = [0, c) for some c > 0. Thus the 
range problem for the ideal window on an interval reduces to the so- 
called a6c-problem for Gabor systems: the classification of all triples 
{a,b,c) of positive numbers such that ^(x[o,c),oZ x Z/6) is a Gabor 
frame [B]. In this paper, we provide a complete answer to the above 
afoc-problem for Gabor systems, see Theorems 2.1 - |2. 5 



By dilation-invariance, Q{x[o,c),ci'^ x Z/6) is a Gabor frame if and 
only if Q{x[Q,dc)ida)'L x Z/((i6)) is for any rf > 0. Thus the a6c-problem 
for Gabor systems can be reduced to finding out all pairs (a, b) of 
time-frequency spacing parameters such that ^(x[o,i),aZ x Z/6) are 
Gabor frames [121 [2S], or all pairs {a,c) of time-spacing and window- 
size parameters such that Q{x[o,c)-,oJj x Z) are Gabor frames. In this 
paper, we state our results without any normalization on any one of 
the time-spacing, frequency-spacing and window-size parameters as it 
does not help us much. 

Our answer to the a6c-problem for Gabor system is illustrated in Fig- 
ure [T| where on the left subfigure we normalize the frequency-spacing 
parameter 6, while on the right subfigure we normalize the window-size 
parameter c and use the conventional frequency-spacing parameter 1/6 
as the |/-axis, c.f. Janssen's tie in [25]. For pairs (a, c) in the red region 
of the left subfigure of Figure fll ^(x[o,c); c^Z x Z) are not Gabor frames 



by Theorem 2.1, while for pairs (a, c) in the green, blue and dark blue 



regions Q{x[a,c)^ci'L x Z) are Gabor frames by Theorems 2.1-2.3 In 
the yellow region, it follows from Conclusion (VI) of Theorem 2.2| that 
the set of pairs (a, c) such that Q{x[q,c)^ 0,2, x Z) are not Gabor frames 
contains needles (line segments) of lengths gcd([cj + l,p)/q — {0, 1/g} 
hanging vertically from the ceiling [cj + 1 at every rational time shift lo- 
cation a = p/q. In the purple region, we obtain from Conclusion (VII) 



of Theorem 2.2 that the set of pairs (a, c) such that Q{x[o,c),ci'^ x Z) 
are not Gabor frames contains floors [cJ > 2 and also needles (line 
segments) of lengths gcd{[c\,p)/q — {0, 1/g} growing vertically from 
floors [cJ at every rational time shift location a = p/q. It has rather 
complicated geometry for the set of pairs (a, c) in the white region such 
that Q{x[o,c),ci'Ij X Z) are not Gabor frames. That set contains some 
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Time spacing parameler a 



Time spacing paiameler a 



Figure 1. Left: Classification of pairs (a,c) such that 
^(X[o,c);^^ X Z) are Gabor frames. Right: Classification of 
pairs (a, 1/6) such that Q{x[o,i)t(^'^ ^ ^/^) ^^^ Gabor frames. 



needles (line segments) on the vertical line growing from rational time 
shift locations and few needle holes (points) on the vertical line located 
at irrational time shifts by Theorems 2.4| and 2^ From the above ob- 
servations, we see that the range of density parameters a, b such that 
the Gabor system Q{xio,c),ci'^ x Z/6) are Gabor frames is neither open 
nor connected, and it has very puzzling structure, c.f. the openness of 
the range of density parameters for a window function in Feichtinger's 
algebra [H]. 

The paper is organized as follows. In Section [2| we state our main 
theorems of this paper. The first two main theorems (Theorems 2.2 



and 2.3) are proved in Sections 3 and 4 respectively. After studying 



various properties of the dynamic system associated with some non- 
contractive and non-measure-preserving transformations in Sections [5} 
we parameterize those triples (a, b, c) of positive numbers such that 
^(X[o,c),a^ X Z/b) are Gabor frames, and finally estabhsh our most 
challenging results (Theorems 2.4 and 2.5) in Sections |6] and [TJ In 
Appendix [A| we provide a finite-step algorithm to verify whether the 
Gabor system Q{x[o.c), ffZ x Z/6) is a Gabor frame for any given triple 
of (a, b, c) of positive numbers. In Appendix IbI we apply our results 
on Gabor systems to identify all intervals I and time-sampling spacing 
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lattices 6Z x oL such that signals / in the shift-invariant space 

AefeZ AebZ 

can be stably recovered from their equally-spaced samples f(to+fi),fi G 
aZ, for any initial sampling position to G M. In Appendix [C| we discuss 
non-ergodicity of a new non-contractive and non-measure-preserving 
transformation. 

In this paper, we will use the following notation. We denote the set of 
rational numbers by Q; the integral part of a real number t by \t\ ; the 
sign of a real number t by sgn(t); the greatest common divisor between 
p and g in a lattice rZ with r > by gcd(p, g); the Lebesgue measure 
of a measurable set E by \E\; the transpose of a matrix (vector) A by 
A"^; the null space of a matrix A by A^(A); the column vector whose 
entries take value r G M by r := (■ ■ ■ , r, r, r, ■ ■ ■ )"^; and the space of 
all square-summable vectors z := (z(A))a6A on a given index set A 
by ^^ := ^^(A), with its standard norm and inner product denoted by 
II ■ II2 := II ■ ||^2(A) and (■, ■) := (-, ■)£2(a) respectively. Also for b > 0, 
we let Bb := {(x(A))Aebz | x(A) G {0, 1} for all A G bZ} consist of all 
binary column vectors whose components taking values either zero or 
one, and B^ := {(x(A))Ae6Z ^ Bb \ x(0) = 1} contain all binary vectors 
taking value one at the origin. 

2. Main Theorems 

In this section, we present our answer to the a&c-problem for Gabor 
systems, and a confirmation to the conjecture in |25l Section 3.3.5]. 

Let us start from recalling some known classification of triples (a, b, c) 
of positive numbers, see for instance [121 UHl [2S] • 

Theorem 2.1. Let {a,b,c) be a triple of positive numbers, and let 



^(X[o,c),aZ X Z/6) be the Gabor system in (1.1) generated by the char- 
acteristic function on the interval [0, c). Then the following statements 
hold. 

(I) If a > c, then Q{x[o.c),0''^ x Z/6) is not a Gabor frame. 
(II) If a = c, then ^(x[o,c),ctZ x Z/6) is a Gabor frame if and only 
if a < b. 

(III) If a < c and b < a, then Q{x[o,c)iOJj x Z/6) is not a Gabor 
frame. 

(IV) If a < c and b> c, then Q{x[o,c),0''^ x Z/6) is a Gabor frame. 



The conclusions in Theorem 12.11 are illustrated in the red and low 
right-triangle green regions of Figure [TJ 
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By Theorem 2.1, it remains to classify triples {a,b,c) of positive 
numbers such that a < b < c. To do so, for any given triple (a, b, c) of 
positive numbers, we let 



(2.1) M,,,,,(t):=(x[o,c)(t-/x + A)) 



fi.eaz,xebz^ 



te 



and we introduce a periodic set 
(2.2) 

T^a,b,c '■= {t G 1^1 Ma^5^c(i)x = 2 for somc binary vectors x G iS^} 

that contains all t on the real line such that there exists a binary vector 
solution X e S^ to the infinite-dimensional linear system 



(2.3) 



Ma,bAt)^ = 2. 



The uniform stability of infinite matrices l\/la,b,c{t),t E M, in (2.1) 



(2.4) < inf inf ||M^,fe,,(t)z||2 < sup sup ||M^,b,,(t)z||2 < oo, 

teM ||2;||2=1 teM ||2||2=1 



was used by Ron and Shen [32] to characterize the frame property for 
the Gabor system Q{x[o,c),o.'^ x Z/6), see Lemma 3.1 in Section[3] 

We observe that infinite matrices l\/la,b,c{t),t G M, in (2.1) have their 



rows containing [c/b\ + {0, 1} consecutive ones, and their rows are 
obtained by shifting one (or zero) unit of the previous row with possible 
reduction or expansion by one unit. In the case that a/b is rational, 
infinite matrices in (2.1) have certain shift-invariance in the sense that 



their (/z + ga)-th row can be obtained by shifting p-units of the /z-th 
row where p and q are coprime integers satisfying a/b = p/q, c.f. [251 
Eq. 3.3.68]. The above observations could be illustrated from examples 
below: 
(2.5) 

/■.::::::: \ 



Ma,b,M 
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for the triple (a, b, c) = ('/r/4, 1, 23— ll7r/2) with irrational ratio between 

a and b; and 

(2.6) 

/■.::::::: \ 



Ma,bAO) 
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V 



for the triple {a,b,c) = (13/17,1,77/17) with rational ratio between 
a and b. Due to the above special structure of the binary infinite 
matrices in (2.1), we may reduce their uniform stability (2.4) to the 



non-existence of trinary vectors in their null spaces, and even further to 
the non-existence of binary vector solutions of the infinite-dimensional 



linear systems (2.3): 



(2.7) G{x[o.c), cfZ X Z/fe) is a Gabor frame if and only if Va^b,c = 0, 
see Theorem 3^ in Section |3| The necessity of the above equivalence 



has been applied implicitly in [191 ES] for their classifications. We apply 



the above equivalence (2.7) to take one step forward in the way to solve 



the afec-problem for Gabor systems. 

Theorem 2.2. Let (a, b, c) be a triple of positive numbers with a < b < 
c, and let Q{x[o,c)-,oJj x Z/6) be the Gabor system in (1.1) generated by 
the characteristic function on the interval [0,c). Set 

Cq = a — [c/b\ b. 

Then the following statements hold. 

(V) {[25\) If Co > a and cq < b — a, then Q{x[Q,c),ciL x Z/6) is a 

Gabor frame. 
(VI) If Co > a and co> b — a, then Q{x[q,c): oZ x Z/6) is not a Gabor 
frame if and only if a/b = p/q for some coprime integers, and 
either 
1) Co > b-gcd{[c/b\+l,p)b/q and gcd{[c/b\ + l,p) 7^ [c/b\ + 
1, or 
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2) cq > b — gcd([c/6J + l,p)b/q + b/q and gcd([c/6J + l,p) = 
[c/b\+l. 

(VII) If Co < a and cq <b — a, then Q{x[q,c): oZ x Z/6) is not a Gabor 
frame if and only if either 

3) Co = 0; or 

4) a/b = p/q for some coprime integers p and q, < Cq < 
gcd{[c/b\,p)b/q and gcd{[c/b\,p) ^ \_clb\; or 

5) ajb = p/q for some coprime integers p and g, < Cq < 
gcd( \c/b\ , p)b/q — b/q and gcd( [c/b\ , p) = [c/b\ . 

The conclusions in Theorem |2.2| are illustrated in the green, yellow 
and purple regions of Figure [Tj 



By Theorems 2.1 and 2.2, we now classify those triples {a,b,c) sat- 
isfying a < b < c and b — a < Cq := c — [c/b\b < a, which turns out 
to b e very complicated. Our approach is not to solve the linear system 
(2.3) directly, and instead to find binary vector solutions x G -B° of a 
"similar" infinite-dimensional linear system 

(2.8) M,,6,,(t)x=l. 

Denote by Sa,b,c the periodic set of all t G M such that there exists a 



binary vector solution to the linear system (2.8), 

(2.9) Sa,b,c ■■= {t e R\ MafiA^)^ = 1 for some vector x G B'^}. 

The set Sa^^c just introduced is a supset of 'Da^b,c in (2.2), and the set 
'Da,b,c can be obtained from Safi,c by some set operations, 

Va,b,c = {Sa,b,c n ([0, Co + a - 6) + aZ) n iSa,b,c - [c/b\ b)) 

U(iSa,fe,c n (UAe[fe,([c/feJ-l)6]nfez('5a,fe,c — A))), 



see (3.27) and Theorem 4.2 Hence the classification of triples {a,b,c) 



of positive numbers such that the Gabor system Q{x[o,c),o,'^ x Z/6) is 
a Gabor frame reduces further to characterizing 

(2.10) i) Sa,b,c = ^, 

and 

(2.11) ii) 5a,b,e 7^ but P,,b,, = 0. 

An advantage of the above approach is that there is a unique solution 
Xt G B^ to the infinite-dimensional linear system l\/la,b,c{t)^t = 1 for 
t G Safi^c, while multiple binary vector solutions could exist for the 
linear system (2.3) for t G Va,b,ci see Proposition 3.9[ Denote by \a,b,c{t) 
the smallest positive index in oL such that ^t{^a,b,c{t)) = 1- This yields 
the following one-to-one map on the set Sa,b,c'- 

(2.12) Sa,b,c 3t^-^t + Xa,bAt) e Sa,b,c, 
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because ta^ i,^(t)Xi G i3° and 

where Ta'Z := (z(A + A'))Aera for z := (z(A))Aera- Further inspection 
shows that the maps t — > t + Xa,b,c(t) and t + Aa^fe^c(^) — )■ t on iSa^t^c 
can be extended to piecewise hnear transformations Ra,b,c and Ra,b,c 
on the real hne M. Here for a given triple (a, 6, c) of positive numbers 
satisfying a < b < c and b — a < cq := c — [c/b\b < a, we define 
piecewise linear transformations Ra,b,c and Ra,b,c on the real line M by 

( t + [c/b\b + b if t G [0, Co + a - 6) + aZ 
(2.13) Ra,b,c{^) = { ^ if t G [co + a - 6, cq) + oL 

\ t + [c/b\ b ate [co, a) + aZ, 

and 

(2.14) 

{t — [c/b\b if t G [c — a, c — Co) + aZ 

t if t G [c — Co, c + 6 — Co — a) + aZ 

t - [c/b\ b-b if t G [c + 6 - Co - a, c) + aZ. 

Our extremely important observation is that Sa^b,c is the maximal in- 
variant set under the piecewise linear transformations Ra,b,c and Ra,b,c, 

Ra,b,cSa,b,c C Sa,b,c and Ra,b,c<Sa,b,c C Safi,c, 

that has empty intersection with their black holes [cq + a — b, cq) + aZ 



and [c — Co, c — Co + fe — a) + aZ, see Theorem 4.1 



Applying the above maximal invariant set property for Sa,b,c, we 
take another step forward in the direction to solve the afoc-problem for 
Gabor systems. 

Theorem 2.3. Let (a, b, c) be a triple of positive numbers with a < b < 
c and b — a < Co := c — [c/b\b < a, and let Q{x[o,c), oZ x Z/6) be the 
Gabor system in (1.1) generated by the characteristic function on the 
interval [0, c). Set 

ci = [c/b\b- [{lc/b\b/a)\a. 

Then the following statements hold. 

(VIII) (lini m\) If [c/b\ = 1, then g{xio,c),aZ x Z/b) is a Gabor 
frame. 
(IX) // \_c/b\ > 2 and c\ > 2a — b, then Q{x[o^c),<^'Z x Z/6) is a Gabor 

frame. 
(X) // \_c/b\ > 2 and C\ = 2a — b, then Q {x[o,c) , ciZ x Z/6) is a Gabor 
frame if and only if a/b = p/q for some coprime integers p and 
q, Cq < b — a + b/q and [c/b\ + 1 = p. 
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(XI) // \c/h\ > 2 and c\ = 0, then Q{x[o,c)iCi^ x Z/&) is a Gahor 
frame if and only if a/h = p/q for some coprime integers p and 
q, Co > a — b/q and [c/b\ = p. 



The conclusions in Theorem 12.31 are illustrated in the blue and dark 
blue regions of Figu re [1} 



By Theorems 2.1 , 2.2 and 2.3 , it remains to classify all triples (a, 6, c) 



of positive numbers satisfying a < b < c, [c/b\ > 2,6 — a < Cq : = 
c- [c/b\b < a and < Ci := [c/b\b- [i[c/b\b/a)\a < 26-a. For that 
purpose, we need explicit construction of the maximal invariant set 
Safi^c if it is nonempty. We observe that Hutchinson's remarkable con- 
struction [22j does not apply as piecewise linear transformations Ra,b,c 
and Ra,b,c are non-contractive. On the other hand, from its maximal 
invariance, we obtain that for the triple (a, b, c) = (7r/4, 1, 23 — ll7r/2) 
with irrational time-frequency-spacing ratio b/a, the maximal invariant 
set 



237r 



77r, 



— '^-y)u[6--,17- 



2l7r, TT 

4 ^4 



'5aAc=[l8-^,ll-y)u[l2 ^ 

has its complement consisting of three holes of size 1 — tc/A on one 
period, and that for the triple (a, b, c) = (13/17, 1, 77/17) with rational 
time-frequency-spacing ratio b/a, 



Sa. 



b,c 



.2 3, r9 10, .12 13, 13^ 

— , — ) U — , — ) U — , — ) H Z 

^17' 17^ ^n'n' hTn' n 



is composed of three intervals of same length 1/17 on the period [0, 13/17), 



see Examples 5A and 5^ A breakthrough of this paper is to show that 
if i^a.fe.c 7^ then the black hole [co + a — 6, Co) + aZ of the transformation 
Ra,b,c attracts the black hole [c — Cq, c — Cq + 6 — a) + aZ of the other 
transformation Ra,b,c when applying Ra,b,c finitely many times, i.e., 

{Ra,b,c)^{[c -Co,c-Co + b-a) + aZ 
for some nonnegative integer L, and hence 

(2.15) Sa,b,c = K\( U^=o (RaAcTilc - Cq, C - Cq + 6 - 

see Theorems 5^ and 5.5[ and Examples 5A, 5^ and 5^ The above 
construction of the maximal invariant set Sa^b,c leads to the following 
characterization of (2.11 ): Sa^b,c 7^ but Va^^c = if and only if{[b/c\ + 
Sa,b,c n [0, Co + a 



[cq + a — b, Co) + aZ 



iZ), 



1 



Tj] + [c/b\ \Safi^c n [co, a) I = a, see Theorem 5.7 



Now it remains to discuss most challenging characterization of (2.10 ): 
classifying all triples {a,b,c) such that Sa^b^c 7^ 0- By (2.15), the maxi- 
mal invariant set Sa^b,c has its complement composed by finitely many 
holes on a period. So we may squeeze out those holes on the line and 
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then reconnect their endpoints. The above holes-removal surgery could 
be described by the map 

(2.16) l^AcW := sgn(t)|[min(0,t),max(0,t)) n5a,fe,c| 

on the line in the sense that it is an isomorphism from the set Sa^b,c 
to the line with marks (image of the holes). In Figure p^ below, we 
illustrate the performance of the holes-removal surgery via 

aJ 3 aexp{27rit/a) i — > Yafi^c{a)ew (-2vriYa,6,c(t)/Xj,b,c(a)) e Ya^bA^^)^^ 

where (f , 1, 23 - i^^), (f,l,f), {§A,^) and (if,l,ff) are used as 



triples {a,b,c) in the four subfigures respectively, c.f. Examples 5.1 



5.2 and 5.3 In that figure, the blue arcs in the big circle are contained 
in aexp(2iTiSa^b,c/0'), the red dashed arcs in the big circle belong to 
aexp{2iTi{M\Sa,b,c)/ci'), and the circled marks in the small circle are 
K,6,c(a) exp (27rzi^a,fe,c/>^a,b,c(a)), where Ka,b,c is the set of all marks on 
the line. 



Having the maximal invariant set Sa,b,c constructed in (2.15) and 



holes- removal surgery described by the map in (2.16), we next consider 
dynamic system of piecewise linear transformations Ra,b,c and Ra,b,c- 
We observe that those piecewise linear transforms are not measure- 
preserving on the real line, but their restrictions on the maximal in- 
variant set Sa^b,c are measure-preserving and one is the inverse of the 
other. More importantly, we show that there is a rotation S{6a,b,c) of 
the circle M/ (Fa;, c (a) Z), 
(2.17) 
S{9a,b,c)iz + ya,bAa)Z) = 6a,b,c + Z + Ya,bAa)Z, z e R/iYa,bAa)Z), 

such that the following diagram commutes, 

<->a,fe,c ^ <~>a,b,c 



(2.18) ^-.M 



Y„„ 



i/{Ya,bAa)Z) > R/(Ya,bA(^)Z) 

S{9a,b,c) 



see Theorem 5.8 In other words, the restriction of piecewise linear 
transformations Ra,b,c and Ra,b,c on the maximal invariant set Sa,b,c can 
be thought as rotations on the circle, see Appendix [C] for the non- 
ergodicity of the piecewise linear transformation Ra,b,c on the real line. 

Having "rotation" property of the piecewise linear transformation 
Ra,b,c, we are almost ready to find all triples (a, b, c) such that Sa^^c 7^ 0- 
We observe that the hole-removal surgery is reversible, that is, the max- 
imal invariant set Sa,b,c can be obtained from the real line by putting 
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£>■ 






0,6 -Oi -0,2 0,2 0,4 0,6 0,8 
(a,b,c)=(i'4,l,23-11i'4) 




-1 -0,0 -0,6 -0,4 -0,2 0,2 0.4 0,6 0.8 1 
(a, b,c)=p/7, 1,23/7) 



/ ' ^ 

1 



^U' 



I 

1 

I / 

I \ 

\ I \ 

\ 1 \ 

\ 1 V 

\ 1 \ 

\ 1 1 



0,6 -0.4 -8,2 8 8,2 8,4 8,6 8,8 
(a,b,c)=(l 3/17, 1,77/17) 




Figure 2. Holes-removal surgery for the maximal invariant 



set Sahci see Examples 5.1, 5.2 and 5.3 



marks at appropriate positions and then inserting holes of appropriate 
sizes at marked positions. But that augmentation operation is much 
more delicate and complicated than the hole-removal surgery. 

For the irrational time-frequency lattice case (i.e., ajh ^ Q), holes to 
be inserted should have the same size h — a (c.f. the upper-left subfigure 
of Figure |2| and the location of marks could be parameterized by the 
numbers of holes contained in the intervals [0, Cq + a — 6) and [cq, a) 



respectively, see Theorem <oA_ This leads to a parametric characteri- 
zation of the statement Sa^h.c 7^ 0, and the following classification of 
triples (a, 6, c) the irrational time-frequency lattice case. 

Theorem 2.4. Let (a, 6, c) he a triple of positive numbers such that 
a < b < c,b — a < Cq := c — [c/b\b < a, [c/b\ > 2, < Ci : = 
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[c/h\h - [{[c/h\h/a)\a <2a-b. Let g{x[o,c),aZ x Z/b) be the Gabor 



system in (1.1) generated by the characteristic function on the interval 
[0,c). 

(XII) If a/b ^ Q, then g{x[Q,c), oIj x Z/6) is not a Gabor frame if and 
only if there exist nonnegative integers di and d2 such that (a) 
a^c-{di + l){ [c/b\ +I){b-a)-{d2 + l) [c/b\ (b-a) e aZ; (b) 
[c/b\b+{di + l){b-a) <c< [c/b\b + b-{d2 + l){b-a); and (c) 
ifEa,b,c = di, where m = {{di + d2 + l)ci-co + {di + l){b-a))/a 
and 

Ea,b,c = {n e [l,di + d2 + l]\ n{ci-m{b- a)) 

(2.19) G [0,co-{di + l){b-a)) + {a-{di + d2 + l){b-a))Z}. 

The conclusion of Theorem 2.4| is illustrated in the white region of 



Figure [Tj In the above theorem, we insert di and d2 holes contained in 
the intervals [0, cq + a — b) and [cq, a) respectively, and put marks at 
U^LY'+Hn(ci - m{b - a)) + (a - (rfi + t/2 + !)(& - a))Z). 

For the rational time-frequency lattice case (i.e., a/b G Q), we 
write a/b = p/q for some coprime integers p and q. Recall that for 
c ^ bZ/q, Q{x[o,c),(iZ x Z/b) is a Gabor frame if and only if both 
GiX[o,[qc/b\b/q),aZ X z/b) and Gix[o,[qc/b+i\b/q),aZ x Z/b) are Gabor 
frames [25l Section 3.3.6.1]. Then it suffices to consider a/b = p/q 
and c/b G Z/q for some coprime integers p and q. In that case, we 
show that marks on the line are equally spaced and gaps to be inserted 
have two different sizes, which could be parameterized by the numbers 
of gaps of large and small sizes contained in intervals [0, Co + a — b) and 
[co,a), see Theorem 7.1 and c.f. Figure p] Applying the characteriza- 



tion for Sa,b,c 7^ in Theorem 7.1, we reach the last step to solve the 
afoc-problem for Gabor systems. 

Theorem 2.5. Let {a,b,c) be a triple of positive numbers such that 
a<b<c,b — a<co<a, [c/b\ > 2 and < ci < 2a — b, where 
Co = c—[c/b\b and ci = [c/b\b—[{[c/b\b/a)\a. LetQ{xio,c),ciZxZ/b) 



be the Gabor system in (1.1) generated by the characteristic function 

on the interval [0,c). 

(XIII) // a/b = p/q for some coprime integers p and q, and c G bZ/q, 
then G{x[o,c), dZ x Z/b) is not a Gabor frame if and only if the 
triple (a, b, c) satisfies one of the following three conditions: 

6) Co < gcd(a,ci) and [c/6J(gcd(a, ci) — cq) 7^ gcd(a, ci). 

7) 6— Co < gcd(a,ci+6) and ([c/6J+l)(gcd(a, ci+6)+co— 6) 7^ 
gcd(a,ci + b). 

8) There exist nonnegative integers ^1,^2,^3,^4 such that (a) 
< a - {di + d2 + l){b - a) e NbZ/q; (b) Nci + {di + 
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ds + 1)(6 - a) G aZ; (c) {di + rfa + l)(A^ci + {di + 4 + 
1)(6 - a)) - (rfi + 4 + l)a G NaZ; (d) gcd(A^ci + (t/i + 
4 + 1)(& - a),Na) = a; (e) i^E^,^^ = d,; (f) cq = {di + 
1)(6 - a) + (rfi + 4 + l)(a - (rfi + ^2 + 1)(& - a))/A^ + -^ 
for some — min(a — cq, (a — (di + (i2 + 1)(& — a))/N) < 
6 < min(co + a — b, {a — {di + d2 + 1)(6 — a))/N); and (g) 
\5\+a/{N[c/h\+{di+d^ + l)) ^ {a-{d^ + d2 + l){a-h))/N, 
where N := di + d2 + d^ + d^ + 2 and E'j^^^ is defined by 



Kb,c = {ne[l,di + d2 + l]\n{Nci + {di + d3 + l){b-a)) 
(2.20) G (0,(^1 + 4 + l)a) + A^aZ}. 



(XIV) ([25]) If a < b < c,b - a < Co < a, [c/b\ > 2, < Ci < 
2a — b, a/b = p/q for some coprime integers p and q, and 
c bij/q, then G{xio,c), oIj x Z/6) is a Gabor frame if and only 
if both Q{x[o,[qc/bib/q), oL X Z/6) and Q{x[o,\<ic/b+i\b/q),ciL x Z/6) 
are Gabor frames. 



The conclusions in the above theorem is illustrated in the white 
region of Figure [I] In Case 6) of Conclusion (XIII) in Theorem 2.5 



the set Kafi^c of marks is (gcd(a, ci) — co)Z and gaps inserted at marked 
positions have same length cq. In Case 7) of Conclusion (XIII) in 



Theorem 2.5, Ka^^c = (gcd(a, ci + b) + Cq — b)Z and gaps inserted at 



marks in Ka^^c are of size b — cq. In Case 8) of Conclusion (XIII) in 



Theorem 2.5, Ka,b,c = hZ,Yafi^c{0') = Nh and gaps inserted at marked 
positions lmh + NhZ, 1 < I < N, have size |6 — a| + |^| for 1 < / < rfi + 
d2 + l and \5\ for ^1+4 + 2 <1<N, where N = di+d2+ds+d4, + 2, h = 
(a- {di + d2 + l)lb-a))/N - \6\,m= {Nci + (rfi + 4 + l)(6-a))/a 
and 6 = Co-{di + l){b-a)-{di + d3 + l){a-{di + d2 + l){b- a))/N. 



Combining Theorems 2.1-2.5 gives a complete answer to the abc- 



problem for Gabor systems. The classification diagram of triples (a, 6, c) 
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in Theorems 2.1 2.5 is presented below: 



(I) a > c 



\ a < c\ 



L (II) a : 



(III) b<a 



a < b < c 



(IV) b>c 



(V) Co > a, Co < b — a 



(VI) CO > a, Co > b — a 



CO < a,co > b — a 



(VII) Co < a,co < b — a 



(VIII) [c/b] = 1 



L [c/b] > 2 ^ 



(XIII) a/b e Z/g, c/6 e Z/q 



(XIV) a/6e Z/q,c/b<^Z/q 



(XII) a/6 ( 



(XI) ci =0_ 



_ < ci < 2a - 6 



a/6 e Q 



(X) ci = 2a -- 6 



(IX) ci > 2a-6. 



From Classifications (V)-(IX) and (XII) in Theorems [Z2}]2^ it 
confirms a conjecture in [2S1 Section 3.3.5]: If a < b < c,a/b ^ Q 
and c ^ aQ + bQ, then Q{x[o,c),cl'^ x Z/6) zs a Gabor frame for L/^. 
This, together with Classification (IV) in Theorem 2.1 and the shift- 
invariance, implies that the range of density parameters a, b such that 
Q{xii '2^ ^ Z/6) is a Gabor frame is a dense subset of the open region 
Wc := {(a, &) : < a < max(6, c)}, where c is the length of the interval 
/. 

3. Gabor Frames and Trivial Null Spaces of Infinite 

Matrices 



In this section, we prove Theorem 2.2 



To prove Theorem |2.2[ we start from recalling some algebraic prop- 
latrices in (2.1): 

^a,b,cit - A')z = M„,fe^c(t)TA'Z for all A' G 6Z 



erties for infinite matrices in (2.1): 

(3.1) 

and 

(3.2) M,,b,c(t - /i')z = ((M,,5,c(t)z)(/i + /^'))^e,2; for all /i' G aZ, 

where t G M, z := (z(A))Aefez, and ryz := (z(A + A'))Aefez- By the shift 
property (3.2) for infinite matrices Mabc(^),^ ^ I^, the sets I'a.fe.c and 
SaM,c in (2.2) and (2.9) respectively are periodic sets with period a, 

(3.3) Vafifi = T^a,b,c + aZ and Sa,b,c = Sa,b,c + aZ. 

Next we recall the equivalence between frame property for the Gabor 
system Q{x[f),c)-,0''^ x Z/6) and uniform stability of infinite matrices 
^a,bfiif)it £ I^, in (2.1), which was estabhshed by Ron and Shen. 
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Lemma 3.1. ([32]) Let (a, b, c) be a triple of positive numbers satisfying 
max (a, 6) < c, Q{x[o,c),ci'^ x Z/6) be the Gabor system in (1.1), and let 
'^a,h,c{t)it £ I^; be the infinite matrices in (2.1). Then Q{x[o,c), oZ x 



Z/6) is a Gabor frame if and only if there exist positive constants A 
and B such that 

(3.4) A||z||2 < ||M„_b,c(t)z||2 < fi||z||2 for all z G £^ and t G M. 



Infinite matrices Ma^;,c(^);^ ^ I^? in (2.1) have their rows containing 



\c/h\ + {0, 1} consecutive ones, and their rows are obtained by shift- 
ing one (or zero) unit of the previous row with possible reduction or 
expansion by one unit, c.f. (2.5) and (2.6). Due to the above special 



structures of infinite matrices in (2.1 ), we can reduce their uniform sta- 



bility property (2.4) to the trivial intersection property between their 
null spaces N {M.a,b,cii)) and the set Bi, — Bb containing trinary vectors 
whose components take values in { — 1,0,1}, and even further to the 
empty-set property for Pa.fe.c- 

Theorem 3.2. Let {a,b,c) be a triple of positive numbers satisfying 
b < c, Q{x[o,c)ioZ, X Z/6) be the Gabor system in (1.1), and let 



a < 



^a,b,ci't),t G M, be infinite matrices in (2.1) 
ments are equivalent to each other. 



Then the following state- 



(ii) 



m 



The Gabor system Q{x[o,c),0''^ x Z/6) is a Gabor frame. 
For every t G M, only zero vector is contained in the intersec- 
tion between Bb — Bb and the null space of the infinite matrix 
M„,fe,c(t); I.e., 

Ni^a.h.cit)) n (Bb - Bb) = {0} for every t G M. 



'a,b,c — V. 

In the next theorem 



it is shown that it could further reduce the 



empty-set property for "Da^b^ in Theorem 3.2 to verifying whether it 
contains some particular points. 

Theorem 3.3. Let (a, 6, c) be a triple of positive numbers satisfying 



a < b < c, the set Va^b,c be as in (2.2), and define 

( {0} if a/b ^ Q, 

(3.5) Za^b '■= I {0,b/q, . . . ,b{p — l)/q} if a/6 = p/g for some 

I coprime integers p and q. 

Then Va,b,c = ^ if and only if'Da,b,c n {Za,b U (c - Za,b)) = 0- 



We remark that the implication (i)^^(iii) in Theorem 3.2 has been 
applied implicitly in [191 ES] for their classification. 



In the next three subsections, we prove Theorem 3.2 2.2 and 3.3 
respectively. 
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3.1. Trivial null spaces of infinite matrices. In this subsection, we 
prove Theorem 3.2 To do so, particularly the implication (iii)^^(i), 
we need an equivalence between empty set property for the set Va^b^c 
and uniform boundedness of maximal lengths of consecutive twos in 
the vector lSAa^b,c{t)^ for any t G M and ^ & Bb- Precisely, we define 

(3.6) Qa,b,c ■= sup Qa.b,c(t) '■= SUp ( SUp Qa,b.c{t, x)) , 

where for any t G M and x G -B?,, 

ir(t,x) := {^ G aZ\ M„,,,,(t)x(/i) = 2}, 

and 

r if ir(t,x) = 

Qa,b,c{t, x) := < sup [n G N| [/x, jj, + no) Pi aZ 

[^ C K{t,ii.) for some /i G aZ} otherwise 

is the maximal length of consecutive twos in the vector Ma,b,c{t)^- We 
show in Lemma 3.4 below that T>afi^c = if and only if the index Qa,b,c 
associated with infinite matrices l\/la,b.c{t),t G M, is finite. 

Lemma 3.4. Let {a,b,c) be a triple of positive numbers satisfying a < 
b < c, and let Nla,b,c{t),t G K; be the infinite matrices in (2.1). Then 
T^a,b,c = ^ if o-nd only if 

(3.7) Qa,b,c < +00. 

IS 



The crucial and most technical step in the proof of Theorem 3.2 
to establish the following stability inequality: 

(3.8) Y. l(M,,,,(t)z)(/i)| > A|z(0)| 

for all t G [0, b) and vectors z = (z(A))AGfez- 

Lemma 3.5. Let {a,b,c) be a triple of positive numbers satisfying a < 
b < c, and let Qa,b,c be as in (3.6). If Qa,b,c < +C)0, then (3.S) holds for 
any t G [0, b) and z = (z(A))Ae6z- 



Now we start the proof of Theorem 3^ by assuming that Lemmas 
and 13.51 hold. 



Proof of Theorem 3.2\ (i)^^(ii): Suppose, on the contrary, that there 
exist to ^ I^) and a nonzero vector z* = (z*(A))Aefez such that 

(3.9) Ma,b,M2* = 0, and z*(A) G {-1,0, 1} for all A G bZ. 
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By Lemma 3.1 and the assumption (i), Ma,b,c{to) has the £ -stabihty; 
i.e., 



(3.10) < inf \\Ma,bAtoM2 < sup ||M„,;,,e(to)z||2 < oo. 

il^il2 = l l|z||2 = l 



This together with (3.9) imphes that 

(3.11) z*^e\bz). 



Set z^ := {z*{X)x[-N,N]{^))x&bz, N > 2. Then we obtain from (2.1), 
(Q and ( [3ll| that 

hm7v_,oo hlfh = 0O7 

||M,,b.,(to)z^||oo < \\Ma,b,MM\o. < c/6 + l,and 

(M,,fe,(to)z^) (/i) = for all fi - to ^ [N - c, N]U [-N - c, -N] . 

Therefore hmjv-^00 ||Ma_b_c(io)z^||2/||z^||2 = 0, which contradicts to the 
£2-stabihty ( p^ . 

(ii)^^(iii): Suppose, on the contrary, that there exist to ^ I^ and 
a vector x G i3° such that Nla,b,c(to)^ = 2. Denote by K the support 
of the vector x; i.e., the set of all A G 6Z with x(A) = 1. Then from 
^a,b,c(to)^ = 2 it follows that 



(3.12) 



#(is: n (-to + /^ + [0, c)) = 2 for all fi G aZ. 



Thus i^{K) = +00 and K = {Xj : j G Z} for some strictly increasing 
sequence {Aj}°2,_o^ in 6Z. For any fi G aZ, we obtain from (3.12) that 
K n (-to + /i + [0,c)) is either {A2J, A2J+1} or {A2j^i, A2j} for some 
j E Z. Thus 

#{Ki n (-to + /i + [0, c)) = 1 for all jj e aZ and i = 0, 1, 

where Ki = {\i+2j ■ J eZ},i = 0, 1. Define x^ := (xj(A))AeaZ, ^ = 0, 1, 
by Xj(A) = 1 if A G -ft'j and Xj(A) = otherwise. Then Xj, i = 0,1, have 
one and only one of them in B^ while the other one in Bb\B'^, and they 
satisfy 

(3.13) X = xo + xi, xo 7^ xi and Ma,b,c(to)^o = Ma,b,c(to)^i = 1- 

Thus y = Xo — Xi is a nonzero vector in Bb — Bb contained in the 
null space of the infinite matrix M.a,b,cito), which contradicts to the 
assumption (ii). 
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(iii)^^(i): Let Qa,b,c be as in (3.6). Then Qa,b,c < oo by Lemma 

[34 For any / G L2, " 



> 



</'6G(X[o,c),aZxZ/fe) 

/_ 3a + 6 + c\ >r^ v;^ v^ 

(q^ac + — ^ — ) E E E 

I / (E^[M(^-/^' + A)/(t + /^ + A))e-2™* 



"dt 



dt 



> 



> 



"E 

4c2 






AebZ 
f6 



E 



E 

0<M'<a<9a,b,c+2a+fe+c AebZ 

\f{t + ^^)\'dt>^^ 



dt 



where the third inequahty follows from Lemma 3^, and the last in- 
equality is true as ^n^azX[o,b)(t — /x) > [b/a\ for all t G M. Therefore 
^(X[o,c)5 oZ X Z/b) is a Gabor frame and the implication (iii)^=^(i) is 
established. D 

We finish this subsection by the proofs of Lemmas 3.4 and 3.5[ 



Proof of Lemma 3.4 ■ {•^=) Suppose on the contrary that T>a^b,c 7^ 0- 
Take to ^ T^a,b,c and a vector xq G B^ satisfying Ma^(,^c(to)xo = 2. Then 
Qa,bfi{i,^f)) = +00, which implies that Qa,b,c(t) = +c>o, a contradiction. 
(^^) Suppose, on the contrary, that Qa,b,c = +00. Then for all n > 
1 there exist t„ G M, /i„ G aZ and x„ G i3b such that Ma,6,c(tn)xn(/i) = 2 
for all fin ^ fi' ^ fin + ^na. By (3.1 ) and (3.2), without loss of generality, 
we may assume that t„ G [0,6) and 

(3.14) (Ma,b,citn)^n){fJ') = 2 for all /i G [— na, raa] fl aZ, 

otherwise replacing t„ by the unique number t'„ G [0,6) satisfying 
tn - fin - na - t'„ G 6Z and x„ by rt/^_t„+^„+„aX„. Moreover, we can 
additionally assume that x„ := (x„(/i))^gfe2 G Bl,n > 1, satisfy 

(3.15) x„/(A) = x„(A) for all A G [—nh,nh] fl 6Z and n' > n, 
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and 
(3.16) 



{tnVr 



is a monotone sequence, 



otherwise, replace {-Xn}'^=i and {t„} 
(3.15) and (3.16). Denote by too 



^1 by their subsequences satisfying 
and Xoo the hmit of the sequence 

{tn}'^=i of numbers in [0, b) and the sequence {x„}^]^ of vectors in B^ 

respectively. Clearly Xoo G -6°. 

If there exists no such that t„ = too for all n > no, then for any given 

for sufficiently large n by (3.15). Thus Ma,b,c(too)^oo 



2 and too ^ 



1^a,b,cj which contradicts to the assumption that Va^^c = v- 

If {tn}^i is a strictly decreasing sequence, then for any given A G 6Z 
and ^ G aZ, X[o,c)(^oo — /^ + A) = X[o,c)(^ n — A* + A) for sufficiently large 
n. This together with (3.14) and ( 3.15[ ) implies that 

(Ma,fe,c(ioo)Xoo)(yU) = lim (Ma,fe,c(in)x„)(/i) = 2, 

n— !>oo 

which contradicts to the assumption that Va^bfi = 0- 

If {tn}'^=i is a strictly increasing sequence, then for any given A G aZ 
and /i G aZ, X(o,c](^oo ~ /^ + '^) = Xfo,c)(^ n ~ ^^ + ^) for sufficiently large 
n. This together with (3.14) and (3.15) yields that 



y^ X(o,c](too - /i + A)xoo(A) = 2 for all /i G aZ, 



Ae6Z 

or equivalently 



y^ X[o,c)(c - too - yu + A)xoo(-A) = 2 for all /x G aZ. 



AebZ 

Thus c — too G 1^a,b,c, which is a contradiction. 

Proof of Lemma 3.5. For t G [0,6), let Aq = 
^0 > be the unique element in [c + /iq — t - 
5q = 0, then (Q holds as |(Ma,fc,c(t)z)(/io)| 
aQa,b,c + 2a + b + c. 



D 



0,;Lto = [t/aja and let 
- 6, c + /io - t) n 6Z. If 
= |z(0)| and /io < t < 



Now we prove (3.8) in the case that ^o > b. To do so, we introduce 
families of triples (X, ni, 6i) e bZ x aZ x bZ,0 < k < Mi,l < I < 6o/b, 
iteratively. For i = 0,1, we let A' = ilb, fi\ = L(t + A')/aJa and Si be the 
unique element in [c + /i' — t — 6, c + /i' — t) fl bZ. Suppose that we have 
defined the triple {X{,fi[, 61), we set Mi = k ii 6l> c + fi[ — t + a — b, 
and otherwise we define (Ai+2./^fc+2' ^+2) by A^^2 = 4 + ^'/^fc+2 = 
L(^ + ^L+2)/«J« and 4+2 e [c + 4+2 - t - 6, c + /i^+2 - ^) ^ 6Z. We 
remark that S^ > c + fj,\ — t + a — b ii the (/i^ + l)-th row of the matrix 
^a,b,c(t) is obtained by shift one unit to the left of the /x^-th row with 
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reduction by one unit, c.f. the fourth and ninth rows in (2.5). Here 



is the visual interpretation of indices X[ and fj,{,l < k < Mi, in the 
matrix Ma,bAt)- 



/^o 



All 



fJ.2 -^ 



IJ-M, 
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From the above construction, we see triples {X[,fi{, 5^), < k < Mi, 
have the following properties: 

. . j \ie[fii-t,f,i-t + a) iiO<k<Mi 

\ ^i+2^[c + f^i-t,c + fii-t + a) iiO<k<Mi-2, 

(3.18) [iJ,M^-t + c,iJ,[^j^-t + c + a)nbZ = ^ if M^ < oo, 
and 

(3.19) {X^f.}j^lQ and {fJf,}j^^Q are strictly increasing sequences. 

Define x; := (x/(A))A6ra by x/(A) = 1 if A = A'^ fo r some < A; < Mi, 
and Xi(A) = otherwise. Then x^ G i3° by (3.19), and for fJ,Q<fJ,< 

(M„,,,,(t)x;)(/i) = { J2 + Yl )x[o,c)(t-/x + Ai) 

0<k<Mi,k even 0<k<Mi,k odd 

= X[o,mj,](/^)+ Y1 ^(ml_„4](/^) 

2<k<Mi,k even 

3<k<Mi,k odd 

where the second equation follows from 

[/^L-2 + a, /^L] C (t + Ai - c, t + X[] C (/ii_2, ^^i + a),2<k<Ml 
which holds by ( |3.17[ ). This leads to the following crucial estimate: 

(3.20) nij^_^ -f^o< aQa,b,c- 
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By (3.17) and (3.19), we have that 

(3.21) fiMi - fJ'Mi^^ < fJ'Aii - fJ'Mi_2 < ^Mi +t- i^Mi -c + t-a) <a + c. 



Combining (3.20) and (3.21) and recalhng no < t < b, we obtain 

(3.22) fiMi < aQa,b,c + a + b + c. 

By ( |3^ , Ml < oo for all 1 < / < 6o/b. Now we establish (Q if 
M;„ is an even integer for some 1 < /q ^ ^o/b. In this case, applying 
(3.17) and (3.18) with I = Iq, we obtain that 

|(M,,,,,(t)z)(4'> )| + |(M„,,,,(t)z)(/i^'], + a) I 
> |(M,,,,,(t)z)(4»,) - (M,,5,c(t)z)(4°, + a) I 

(3.23) = |z(A'2V2)-z(A^y| 

for all integers < k < M/,,/2 — 1, and 

(3.24)|(M,,,,,(t)z)(/i^",)|- 



M,,,,,(t)z)(/i'4 + a)|>|z(A'" 



for k = Mij2. Combining ( |3.22[ ), ( |3.23[ ) and ( |3.24[ ), we get that 
2 Yl |(M,,,,,(t)z)(/x)| 

M,o/2 

> Yl l(Ma,v(t)z)(/i^°J| + |(M,,,,e(t)z)(/i^l, + a)\ 

k=0 

Mio/2-l 

fc=0 



Hence (3.8) follows in the case that Mig is even for some 1 < Iq < So/b. 
Finally we prove (3.8) in the case that Mi, 1 < / < So/b, are all odd 
integers. In this case, mimicking the argument used to establish (3.23) 
and ( |3.24 ), we obtain that 
(3.25) 

for all integers < A; < (M; — 3)/2, and 

(3.26) \{Ma,tAt)zM,+,)\ + \{M,,tAt>Mk+i + a)| > |z(A^,J| 
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for k = {Mi- l)/2, where 1 < / < 6o/b. Therefore 
26o 



^ Y, |(M„,,,,(t)z)(/x)| 

0<M<aQa,b,c+2a+fe+c 

(M,-l)/2 

> |(M,,,,,(t)z)(^o)|+ Yl E 

l<K5o/b A.-0 

+ |(M„,5,c(t)z)(^^,+i + a) 

0<A<<5o l<K5o/b 

= I E ^(^)|+ E 1^(^)1 >i^ 

0<A<<5o l<A<<5o 



{M,,bAt)^W2k+i)\ 



by (3.25) and (3.26). This together with (5o < c + Ho — t < c proves (3.8) 
in the case that Mi,l < I < Sq/B, are all odd integers. This completes 
the proof of Lemma 3.5 D 

3.2. Proof of Theorem 2.2[ In this subsection, we apply Theorem 



3.2 to prove Theorem 2.2 by verifying whether Va^^c is an empty set 
or not. To do so, we notice that Sa,b,c is a supset of Va^^c-i 



(3.27) 



T^a,b,c C Sa,b,i 



because any vector x G i3° satisfying 'M.afi,cif)^ = 2 can be writ- 
ten as the sum of two vectors Xi,X2 G Bb such that 'M.a,b,c{i)^i = 
^a,b,cit)^2 = 1, c.f. (3.13) in the proof of Theorem 3.2 We also we 
need some elementary properties for the supset Sa,b,c, including empty 
intersection property (3.30) with the black holes. 

Proposition 3.6. Let {a,b,c) be a triple of positive numbers with a < 
b < c. Set Co = c — [c/b\b, (cq + a — 6)+ = max(co + a — b,0) and 
Co A a = min(co, a). Define 



(3.28) Xa,bAt) 



[c/b\b + b if t G [0, (co + a-6)+) + aZ 

if t G [(cq + a — 6)+, Co A a) + aZ 

[c/b\ b if t G [co A a, a) + aZ, 



and 
(3.29) 

Xa,b,c(t) 



- [c/b\ b-b if t G [c - (co + a - 6)+, c) + aZ 

if t G [c — Co A a, c — (co + a — 6)+) + aZ 

— \_c/b\ b if t G [c — a, c — Co A a) + aZ. 
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Then 

(3.30) 5a,fe,cn([(co+a-6)+,coAa)U[c-coAa, c-(co+a-6)+)+aZ) 

and 

(nn^\ ( \\ _ j ^ if Aa,fe,c(t) < A < A^^fc^cW and A 7^ 

^ ^ ^ ^"l 1 ifA = A,,Ut),0,A,,U^) 

for any t G Sa,h,c and ^E B^ satisfying 'M.a,b,cif)^ = 1- 



Now assuming that Proposition 3.6 holds, we start to prove Theorem 
2.2[ The main ideas behind our proof of Theorem|2.2|are as follows. To 



prove Conclusion (V) of Theorem 2.2, we use Proposition 3.6| to verify 
that Sabc = 0; which together with (3.27) and Theorem 3.2 leads to 



the desired frame property for the Gabor system ^(x[o,c)aZ x Z/6). The 
crucial step in the proof of Conclusion (VI) is that for any to ^ T^a,b,c, 
the binary vector ^s. E B'^ satisfying A4a,b,c(to)^ = 2 is supported on 
( [.c/b\ + 1)6ZU (Ai + ( [c/b\ + l)bZ) for some Ai € [b, [c/b\b]nbZ, which 
implies that 

{to, to + A} + ( [c/b\ + l)bZ + aZc Va,b,c, 



see (3.33) and (3.34). Define 



(3.32) Ma,6,c(t) = (X(o,c] (t - /i + \))^eaz,xebz, t E M. 

We use similar argument to prove Conclusion (VII), except that for 
any to ^ i'a,b,c '■= c — 'Da^b,c, the binary vector :s. E B'^ satisfying 
■M.a,b,c(to)^ = 2 is supported on [c/b\bZ U (Ai + [c/b\bZ) for some 
Ai E \b, [c/b\b- 1] n bZ, see ( |339| and ( |3lo| ). 



+ aZ = M in this case. This conclusion was 
Section 3.3.3.2], we include the proof as the 



will be used later to prove Theorem 3.3 



Proof of Theorem \2.^ (V): By (3.27) and Theorem 3.2, it suffices 
to prove Safi^c = 0, which follows from Proposition 3.6 as [(co + a — 
6)+, Co A a) + aZ = [0, a) 
also established in 
conclusion Sa,b,c 

(VI): {=^) By Theorem [Sg Va,b,c ^ 0- Then by ( |3^ th ere exist 
to E [0,a) and ^ E B^ such that Ma(„c(to)x = 2. By (3.13), we can 
write X = xq + xi such that xq E B^.^i E Bb and Mabc(to)xo = 
N\.a,b,c(to)^i = 1- Let \i,i = 1,2,3, be the first three positive indices 
in 6Z with x(Aj) = l,i = 1,2,3. Then A2 is the first positive index in 
bZ with Xo(A2) = 1 by (3.13). By Proposition 3.6, we obtain that A2 = 
A3 — Ai = [c/b\b + b. Further inspection also shows that to E [0,Co + a — 
6) , to + Ai E [0, Co + a — fe) + aZ, and the unique number ti E [0, a) fl (to + 
A2 + aZ) belongs to 'Da^b,c, where < Cq := c — [c/b\b < b. Inductively 
for any n > 1, the unique number t„ in [0, a) fl (t„_i + ( [c/b\ + l)b + aZ) 
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belongs to "Pa.b.c n[0, co + a — b) and tn + Xi G ^a,(),cn([0, co + a — 6)+aZ). 
From the above construction, we see that tn — to — n( [c/b\ + 1)6 G aZ. 
Hence a/b is rational, as otherwise {tn '■ n > 0} is dense in [0, a), which 
is a contradiction as co + a — b < a and t„ + Ai G [0, cq + a — 6) + aZ for 
all n > 0. Now we write a/b = p/q for some coprime integers p and q, 
set m = gcd( [c/b\ +l,p), and let t^ be the unique number in [0, mb/q) 
such that tg — to G mblj/q. Then 

{t„:n>0} + aZ = {to + ri([c/6J + 1)6 : n > 0} + aZ 

= to + ( [c/6J + 1)6Z + aZ = to + mbZ/q 

(3.33) = {tp + nmb/q : < n < p/m — 1} + aZ, 

where the first equality follows from the definition of t„, the second 
one holds as p{[c/b\ + 1)6 G aZ, and the last one is true by m = 
gcd(([c/6J + 1)^,^). Similarly, we have that 

(3.34) {t„ + Ai :n > 0} + aZ = {tQ+nmb/q: < n < p/m-l} + aZ, 
where t'^ is the unique number in [0, mb/q) such that tg — to — Ai G 



mb'L/q. Hence from (3.33), (3.34) and the property that tn.tn + Ai G 
[0, Cq + a — 6) + aZ, n > 1 , we obtain that {tg + nmb/q : < n < p/m — 
1} C [0,Co + a-6) and {to + nm6/g : < n < p/m-1} C [0,Co + a-6). 
Observe that t'^ - tg G Ai + m6Z/g C 6Z/g, and that tg' - t'o ^ if 
m = [c/6J + 1 as otherwise (Ai/6)g = ( [c/6J + l)r for some r G Z. This 
together with 1 = gcd{q,p) > gcd(g, m) implies that Ai G ([c/6J + l)6Z, 
which is a contradiction as < Ai < ([c/6J +1)6. Therefore Co + a — 6 > 
{p/m, — l)mb/q ii m ^ lc/b\ + 1 and Co + a — 6 > {p/m — l)mb/q + b/q 
ii m, = \_c/b\ + 1. This completes the proof of the necessity. 

(<^^) In the case that gcd([c/6J+l,p) ^ [c/6J + l, we define x(A) = 1 
if A G {0,gcd([c/6J + l,p)b} + ([c/6J + 1)6Z and x(A) = otherwise. 
Then ^ E B'^ and 

M,,fe,,(0)x(/x) = J2x[o,c){k{lc/b\+l)b-fi) 

fcez 

+ Yl ^[0'-) (^( Lc/&J + 1)& + gcd( [c/b\ + 1, p)6 - /i) 
fcez 

2 

= $^$^X[o,c)(A:(Lc/6J +l)6 + s,gcd(Lc/6J +l,p)6/g) 

2 

(3.35) = ^X[o,c)(sigcd([c/6J + l,p)b/q) = 2, /i G aZ, 

where < Si,S2 < 9(Lc/6J + l)/gcd([c/6J + l,p) — 1 are so chosen 
that -/x-Sigcd(Lc/6J + l,p)6/g G ([c/6J +l)6Zand -/x + gcd([c/6J + 
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l,p)b - S2gcd(Lc/6J + l,p)b/q G ([c/6J + l)bZ. Therefore Va,b,c ^ 
and hence the corresponding Gabor system Q{x[o,c),ci'^ x Z/6) is not a 
Gabor frame by Theorem |3.2[ 

In the case that gcd([c/6J + l,p) = [c/b\ + 1, we have that p > 2 
and we define x(A) = 1 if A G {0, Ai} + {[c/b\ + l)bZ and x(A) = 
otherwise, where Ai G 6Z is chosen so that b < Ai < {p — l)b and 
qXi/b — 1 G pZ. Then 

M,,b,e(0)x(/i) = Y,X[o,c){k{[c/b\+l)b-fi) 



kez 
J2xm{H[c/b\+l)b + s,{[c/b\+l)b/q) 



+ Yl ^[O''^) (^( Lc/feJ + 1)& + S4( Lc/&J + l)b/q + b/q) 



(3.36) 



+Xm{^4{[c/b\ + l)b/q + b/q) = 2 



where < S3, S4 < g — 1 are so chosen that —fi — S3([c/6J + l)b/q G 
{[c/b\ + l)bZ and -/i + Ai - 6/^ - S4{[c/b\ + l)b/q G ([c/6J + l)bZ, 
and the last equahty follows from the assumption on cq. Therefore 
1^a,b,c 7^ in the case that gcd([c/6J + l,p) = [c/b\ + 1 and hence the 



Gabor system G{x[o,c)^ o,'^ x Z/b) is not a Gabor frame by Theorem 3.2 
(VII): Let Ma,b,c(t), t G M, be as in ( |3.32[ ) and define 

(3.37) Pa.b.c = C - Va,b,c- 

Then 

(3.38) Pa,b,c = {t G M : Ma,bAt)^ = 2 for some x G i3|^}. 

We will apply an argument similar to the one used in the proof of the 
conclusion (VI) essentially replacing Aia,b,c(t), T^a,b,c and [c/b\ + 1 by 
Ma,b,c{t)^ i^aM and. [c/b\ respectively. 

(^^) Let to ^ '^a,b,c for some to ^ (0,a]. The existence of such a 



num ber to follows from (3.1), (3.37) and Theorem 3.2 Then by (3.32) 
and {3^38)j_Ma,b,c(to)x = 2 for some x G B'^. Following the argument 
to prove ( |3.13 ), we can write x = Xq + Xi for some Xq G B^ and Xi G Bb 
satisfying Wla,b,c(to)^o = ^a,b,c(to)^i = 1, and let Xi,i = 1,2,3, be the 
first three positive indices in bZ with x(Aj) = l,i = 1,2,3. Then A2 
is the first positive index in bZ with xo(A2) = 1, and A2 = A3 — Ai = 



[c/b\b by following the argument used in the proof of Proposition 3.6 
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Furthermore to ^ (co, a],to + \i G (cq, a] + aZ, and the unique number 
ti G (0, a] with ti — to — ^2 G aZ belongs to Va,b,c- Inductively for any 
n > 1, the unique number t„ G (0, a] satisfying t„ — t„„i — [c/6J6 G aZ 
belongs to r'a^^ c fl (co, a] and t„ + Ai G 2^a,fe,c H ((co, a] + aZ). From the 
above construction, t„ — to — n[c/6J6 G aZ. Hence a/h G Q if Co 7^ 0, 
as otherwise {t„ : n > 0} is dense in (0,a], which is a contradiction. 
Now consider Co 7^ and write a/h = p/q for some coprime integers p 
and g, set m = gcd([c/6J,p). Let tg and tg be the unique number in 
(0, mb/q] such that tg — to G mblj/q and tg — to — Ai G mhlj/q. Then 

(3.39) {tn : n > 0} + aZ = {tg + nmfo/g : < n < p/m - 1} + aZ, 
and 

(3.40) {t„ + Ai :n > 0} + aZ = {t'o+nrnfo/g: <n <p/m-l} + aZ. 



Hence from ( |3.39[ ), (3.40) and the property that tn.tn + Ai G (co,a] + 



aZ, ra > 1, it follows that {tg + nmb/q : < n < p/m — 1} C (co,a] 
and {tp + nmb/q : < n < p/m. — 1} C (cq, a]. Observe that tg — tg G 
Ai + mblj/q C 6Z/g and that t'o — tg 7^ if m = [c/6J as otherwise 
Ai G ?7i6Z, which is a contradiction. Therefore < cq < mb/q if 
m 7^ Lc/6J and < co < m6/g — b/q if m = \c/b\ . This completes the 
proof of the necessity. 

{'^=) In the case that Cq = 0, we define x(A) = 1 if A G {0,6} + 
[c/6J6Z and x(A) = otherwise. Then x G i3j^ and 

Ma,6,c(0)x(/i) = ^ X[o,c) (-Ai + ^c) + X[o,c) ("/i + 6 + A;c) = 2 

fcez 

for all fl G aZ. Then Va^b,c 7^ and ^(x[o,c); c^^ x Z/6) is not a Gabor 
frame by Theorem |3.2[ 

In the case that a/b = p/q for some coprime integers p and q, < 
Co < gcd{[c/b\,p)b/q and gcd([c/6J,p) 7^ [c/b\, we define x(A) = 1 if 
A G {0,gcd([c/6J,p)6}+ [c/6J6Z and x(A) = otherwise. Then x G B^ 
and 

M„,fe,c(0)x(/i) = J2x{o,c]isigcd{[c/b\,p)b/q + k[c/b\b) 

kez 

(3.41) +^X(o,c](s2gcd(Lc/6J,p)6/? + fcLc/&J6) = 2, 

where 1 < si,S2 < q[c/b\/gcd{[c/b\,p) are so chosen that —fx — 
Sigcd( [c/b\ , p)b/q G [c/b\ bZ and — /i+gcd( [c/6J , J9)6— S2gcd(_[c/6J , p)b/q G 



[c/6J6Z. Hence G I'a.b.c (or equivalently c G I'a.fe.c by (^3.38)) and 



S{X[o,c), oIj X Z/6) is not a Gabor frame by Theorem 3.2 
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In the case that a/h = p/q for some coprime integers p and q, < 
Co < gcd{[c/b\,p)b/q — b/q and gcd([c/6J,p) = [c/b\, we have that 
[c/b\ > 2 and define x(A) = 1 if A G {0, Ai} + [c/b\bZ and x(A) = 
otherwise, where & < Ai < ([c/&J — l)b is so chosen that q\i/b + 1 G 
[c/b\Z. ThenxG B^ and 



MaAc(0)x(/i) 



(3.42) 



Yl Mo,c] (S3gcd( [c/b\ , p)b/q + k [c/b\ b) 
kez 

+ Yl ^(0.-1 ^^4gcd( [c/b\ , p)b/q -b/q + k [c/b\ b) = 2, 



where 1 < S3,S4 < q[c/b\/gcd{[c/b\,p) are so chosen that — /i — 
S3gcd( [c/b\ , p)b/q G \c/b\ b7L and — /i + Ai — bjq — S4gcd( [c/6J , p)b[qE 
[c/6J6Z. This proves that G T>a^i,c and hence c G I'a.fe.c by (3.38). 



Thus Q{x[o,c)i ci^ X Z/&) is not a Gabor frame by Theorem 3.2 D 



We finish this subsection with the proof of Proposition 3.6 



Proof of Proposition 3. 6. Let t G Sa^b,c and ^a,b,c{t)'x. = 1 for some 



X G B^. By (3.2), we may assume that t G [0,a). Let Ai G bZ be the 



smallest positive index such that x(Ai) = 1. Then Ai > [c/b\b because 
(3.43) 

1 = X[o,c)(t) < X[o,c)(i) + xio,c){t + ^i) < 5^ X[o,c)(t + A)x(A) = 1, 

and Ai < [c/6J 6+6 since otherwise ^;^g^^X[o,c)(^—o+A)x(A) =0. Thus 
Ai is either [c/6J6 or |_c/6J6 + 6. If Ai = [c/6J6, then t > cq (and hence 
min(co, a) <t < a) hy (3.43). If Ai = [c/6J6 + b, then t < Cq + a — 6 as 
1 = E A6bz X[o,c) (t - a + A)x( A) = X[o,c) (t - a + [c/b\ b + b). This proves 
that t ^ [(co + a — 6)+, min(co, a)) and Ai = \a,b,c(t)- Hence 



(3.44) 
and 

(3.45) 



<Sa,b,c n ([(co + a - 6)+, Co A a) + aZ) 



x(A) 



if < A < A„,6,e(t), 

1 iiX = Xa,bAt)- 



For the above vector x, we have that lSAa^b,cic — t)x = 1, where 
Ma,6,c(i) is given in (3.32) and x = (x(— A))A6bz ^ B'^. Then mimicking 
the above argument to establish (3.44) and (3.45), we obtain that 

(3.46) Sa,b,c n ([c - Co A a, c - (co + a - 6)+) + aZ) = 0, 

and 



(3.47) 



x(A) 



if < A < -\a,bAt) 

1 if X = -~Xa,bAt)- 
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Combining (3.44)-(3.47), we obtain (3.30) and (3.31). 



D 



3.3. Binary solutions of infinite-dimensional linear systems. In 



this subsection, we prove Theorem 3^ To do so, we first recall some 
basic properties concerning black holes, ranges, invertibility of piece- 
wise linear transformations Ra,b,c and Ra,b,c on the real line. 

Proposition 3.7. Let {a,b,c) be a triple of positive numbers satisfying 
a < b < c and b — a < Cq := c — [c/b\b < a, and let Rafi,c cmd Ra,b,c be 
infinite matrices in (2.13) and (2. 14)- Then 



(3.48) 
(3.49) 



Ra,b,c{t) = t a t E [cq + a — b, Co) + aTL 
Ra,b,c{^) = t if t G [c — Co, c + 6 — Co — a) + aZ, 



Ra,b,c{^\{[co + a-b,co) + aZ)) = M\([c - cq, c + 5 - co - a) + aZ) 
Ra,b,c{^\i[c - Co, c + 6 - Co - a) + aZ)) = R\{[co + a - 6, co) + aZ), 

and 

(3.50) 

Ra,bA^a,bA^)) = t for all t ^ [co + a - b, cq) + aZ 
Ra,b,c{Ra,b,cif)) = t for all t ^ [c — Co, c + 6 — Co — ct) + aZ. 



The black hole property (3.48), the range property (3.49), and the 



invertibility property (3.50) outside black holes follow directly from 



(2.13) and (2.14). We omit the detailed arguments here. 



To prove Theorem |3.3[ we then show that the piecewise linear trans- 
formations Ra,b,c and Ra,b,c have their restrictions on the sets Va^hc and 
Sa,b,c being well-defined, invariant, invertible and measure-preserving. 

Proposition 3.8. Let {a,b,c) be a triple of positive numbers satisfying 
a < b < c and b -_a< Cq := c— [c/b\b < a, the sets 'Da^b,c o^nd SaM,c be 
as in (2.2) and (2.9), and let transformations Ra,b,c and Ra,b,c be as in 
( 2.13^ and (2.14). Then the following statements hold. 

(i) The sets 'Da^b,c CLnd Sa,b,c (i,re invariant under the transforma- 
tions Rabc O'TT'd Rabci i- C , 

(3-51) 

-'^a,b,c^a,b,c J^a,b,c'^a,b,c ^a,b,c aUQ -ria,b,c'~^a,b,c J^a,b,c^a,b,c ^a,b,c- 

(ii) The restriction of Ra,b,c onto Sa,b,c (resp. Va^^^c) is the inverse of 
the restriction of Ra,b,c onto Sa,b,c (resp. T>a,b,c), and vice versa; 
i.e., 
(3.52) 

Ra,b,c{RaAc{t)) = RaAc{Ra,b,c{t)) = t for all t € Sa^},^c (reSp. Pa.fe.c)- 
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(3.53) 



The restriction of the piecewise linear transformations Ra,b,c O'^d 
Ra,b,c onto Sa,h,c (resp. 'Da^b,c) CLre measure-preserving transfor- 
mations; i.e., 

\Ra,b,c{E)\ = \Ra,bAE)\ = l-^l 

for any measurable set E C Sa,b,c (resp. Va^^. 



We will use Proposition |3.8| to establish the uniqueness of solution 
yi E B^ for the infinite-dimensional linear system 'M.a,b,c{t)^ = 1, which 
is a pivotal observation and our starting point to explore various prop- 
erties of the sets Va^b^c and Sa,b,c deeply. 

Proposition 3.9. Let {a,b,c) be a triple of positive numbers satisfying 
a < b < c and b — a < cq := c — [c/b\b < a, and let the set Sa,b,c be 
as in (2.9). Then for any t G Sa,b,c there exists a unique vector x G ;B° 
such that ls/ia,b,c{t)yi = 1. 



We will also apply Proposition |3.8| to show that the black hole of 
the piecewise linear transformation Ra,b,c has empty intersection with 
the invariant set Sa^b,c, which plays important roles in the explicit con- 



struction of the maximal invariant set iS„ /,^ in Theorems 5.4 and 5.5 



Proposition 3.10. Let {a,b,c) be a triple of positive numbers satisfy- 
ing a < b < c and b — a < Cq := c — [c/b\b < a, the set Sa^^c be as in 



(2.9), and let the transformation Ra,b,c be as in (2.13). Then 
(3.54) {R,^f„c)'"{[c-Co,c-Co + b-a)+aZ)nSa,b,c = 



for all n > 0. 



To prove Theorem 3^, we finally need the following density property 
of the sets Safi,c and Va^b^c around the origin, c.f. Lemma 5.12 for 



similar density property of the set Safi,c when the ratio between a and 
b is rational. 

Lemma 3.11. Let {a,b,c) be a triple of positive numbers satisfying 
a < b < c,b — a < Co := c — [c/b\b < a and a/b ^ Q, the piecewise 



linear transformation Ra,b,c be as in (2.13), and let Va,b,c O'^d Sa,b,c be 



as in (2.2) and (2.9) respectively. Then the following conclusions hold. 



(3.55) 



(3.56) 



If <Sa,b,c 7^ 0, then 

(0, e) n Sa,b,c ^ and (-e, 0) n Sa,b,c ^ ( 

for any e > 0. 

IfVa,b,c7^^, then 

(0, e) n Va,b,c ^ and (-e, 0) n Va,b,c y^ 
for any e > 0. 



32 



XIN-RONG DAI AND QIYU SUN 



We prove Theorem 3^ by assuming that Proposition 3^ and Lemma 
HHlhold. 



Proof of Theorem \3.3[ The necessity is obvious. 

Now the sufficiency. By (Q, Ks^, KM, IKA^, KM, and 



Theorems |2.1| and |2.2| it remains to prove the sufficiency for the cases 
that b — a < Cq < a and a/b ^ Q and that b — a < cq < a and a/b G Q. 

Case 1: b — a < Cq < a and a/b Q. 

Suppose on the contrary that 'Da^b,c 7^ 
3.11, and the assumption that Va^b,c^2^a,b = 



Then by (3.2), Lemma 



5, there exist t 
(0, a),n > 1 such that {in}^i is a decreasing sequence that converges 



to zero. Following the argument used in the proof of Lemma 3^, we 
obtain that G Va,b,c- This leads to the contradiction. 

Case 2: b — a < cq < a and a/b G Q. 

Write a/b = p/q for some coprime integers p and q. Suppose on the 
contrary that Va^^c 7^ 0; i-e., Ma,b,cit)^ = 2 for some t G M and x G i3°. 
By (3.2) and the assumption that 'Da^b,c H Za^b = 0, we may assume 



that t G [0,pb/q)\{0, b/q, . . . , b{p — l)/q} without loss of generality. If 
^ t - [qt/b\b/q < c- [qc/b\b/q, then X[o,c){.[qt/b\b/q - /i + A) = 
X[o,c){t — /i + A) for all yU G aZ and A G 6Z. This implies that 

Ma,v(Lg^/^J Vg)x = M„,b,,(t)x = 2, 
and hence \_qt/b\b/q G X'atc H Zq^, which is a contradiction. 

If ^ t - \(lt/b\b/q >'c - Lgc/6J6/g, then X(o,c]((L9V&J + 1)V? " 
/i + A) = X[o,c)(^ — /i + A) for all ji E aX and A G 6Z. Hence Ma^b^c(c — 
([gt/6J + l)6/g)x = 2, where x = (x(— A))Aebz ^ -6°. This contradicts 
to the assumption that 'Da,b,c ^{c — Za^) =0. □ 



We finish this subsection with proofs of Proposition |3.8[ Lemma [3. 11 
and Propositions 3^ and |3.10 



Proof of Proposition 3.8 
that 



Ra,b,cSa,b,c C Sa. 



(3.57) 
and 
(3.58) 
Observe that 

(3.59) Ra,b,c{Ra,b,c(t)) = Ra,b,c{R> 

by Propositions |3.6| and |3.7[ Hence 



i) By (3.1) and Proposition 3.6, we have 

J^a,b,c'~^a,h,c ^ '~^a,b,c 



b,c- 



,(t)) = t for all t G Sa,b, 



(3.60) 



J^a,b,c'~^a,b,c J^a,b,c'~^a,b,c '~^a,b,c 



by (3.57), (3.58) and (3.59). 
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Let t G Va,b,c and Ma,6,c(t)x = 2 for some x G B'^. By ( |3.13[ ), 
there is a decomposition x = xq + xi such that xq G i^^jXi G Sft and 
Ma,b,c{'t)^o = Ma,bAt)^i = 1- Therefore r^^ ,^(i)X G i3|^ by Proposition 
3^ and Ma,bA^a,b,ci't))^K,b,M^ = Ma^bA^)^ = 2- This proves that 

(3.61) Ra,b,cT^a,b,c C Vafi^c- 

Similarly we have that 

(3.62) Ra,b,cT^a,b,c C. Vafi^c- 

Recalling that Va,i,c C Sa,b,c and combining (3.59), (3.61) and (3.62), 
we obtain 

(3.63) Ra,b,c^a,b,c = Ra,b,c^a,b,c = '^a,b,c- 



Therefore (3.51) follows from (3.60) and (3.63). 

(ii) The invertibility of the transformations Ra,b,c and Ra,b,c in the 
second conclusion follows from (3.30) and (3.50). 

(iii) 



By (2.13) 



(3.64) \Ra,bAE)\ < \E\ 

for any measurable set E, and the above inequality becomes an equality, 

(3.65) \Ra,bAE)\ = \E\, 

whenever E has empty intersection with the black hole [cq + a — b, cq) + 



aZ of the transformation Ra,b,c- This, together with (3.27), (3.30) and 



the first conclusion, proves that Ra,b,c is a measure-preserving transfor- 
mation on the sets T>a^b,c and Sa^b,c- 

The measure-preserving property for the transformation Ra,b,c can 
be established by applying similar argument. D 



Proof of Lemma 3.11 . (i): Let t„ := {Ra,b,c)"' i^o)^ n > 0, be the orbit 
of the transformation Ra,b,c with initial to G Sa,b,c, and set t„ := tn — 
[tn/ci\a,n > 0. Without loss of generality, we assume that t„ 7^ for 
all n>0, because at most one in the sequence in,n > 0, could be zero 
by the assumption a/b ^ Q and in that case we replace the initial to 
by tno for some sufficiently large uq. Clearly the proof of the first non- 



empty-set conclusion in (3.55) reduces to proving that for sufficiently 



small e > there exists an index n with the property that 
(3.66) t„e(0,e). 

and 



From (3.3), Propositions 



3.6 



3.8 



it follows that t„ G i5a,fe,cn([0, Co + 
b) n [cq, a)),n G Z+. Recall that for n ^ m, tn — tm = kb for some 



7^ /c G Z by (2.13), which together with a/b ^ Q implies that 



(3.67) 



tn — imi^^ whenever n ^ m. 
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As tn,n > 0, lie in the bounded set (0,a), there exist integers rii < 722 

such that |t„j — in2\ < e- Without loss of gen erality, we assume that 

im ^ (0,e 

111. Recall that t„ 

that either i^ , t„2 



otherwise the conclusion (3.66) holds by letting n 
[0, Co + > 



h) n [co, a) for all n > 0, we have 
- h) or ini,in2 £ [co,ci). This im- 



[0, Co + a 

plies that \a,b,c(tm) = K,b,c{tn2) 

If K,b,c{tni+k) = K,b,c{tn2+k) for all positive integers k, then t„^+fc 



where \a,b,c{t) is defined in (3.28). 



67- 



tni —tn2 for all positive integers k. Applying the above equality 



with k = {n2 — ni)l,l G N, we obtain that t„^ 



for all I e N. Therefore tn^+j^^{n2-ni) 

for to = L^ni/(^ni ~ ^n2)\ il ^n2 ~ ^m 



+I{n2—ni) ''ni~rH^"2 »^iJ 

^m + lo(tn2 - inj G (0,e) 



< 0, and t 



ni+{li+l){n2~ni) 



tni + {h + l)(tn2 - tni) - fl G (0, e) f or h = [{a - tnM jino - t„JJ 



if t„2 ~ ^m > 0- This together with (3.67) proves that (3.66) holds 



when Xa.b.citn^+k) = Xa,b,citn2+k) for all positive integers k. Now we 
consider the case that there exists a positive integer m such that 
Aa,6,c(^n.i+fc) = Aa,6,c(in2+fc) for all uouuegative integer k e [0, m - 1] 
and Xa,b,citni+m) 7^ Aa,fe,c(^n2+m)- Then we can prove by induction that 
tr,,-\-k — tn2+k = tni — tn2 for all < A; < m — 1. This implies that either 



'^rij+k 
\^ni+m- 



-1 f-n; 



+m-l\ 



I'^ni '^n2 1 



or |t„^ 



+m-l 



-t 



n2+in—l 



~r \^n\ ^' 



"2 1 



On the other hand, it follows from \a,b,c{tni+m) 7^ Xa,b,c{t 



a,b,c\'^n2+m) 



that 



t 



ni+m— 1 



) '■n; 



+m— 1 



should lie in the different intervals [0, co + a — 6) and 



[co,a). Therefore |t„^ 



+m-l 



-t 



n2+m—l 



~r l^-ni '^ 



"2 1 



which implies 



that one and only one of t„^+(m-i), t„2+(m-i) belongs to (0, e), while the 
other one of t„^+(m-i),^n2+(m-i) belongs to [a — e, a) . Hence (3.66) 

holds when Xa,b,ci'tni+m) 



Xa,b,c{t 



n2+mj 



for some positive integers m 



and we complete the proof of the conclusion (3.66). 



The second conclusion in (3.55 ) can proved by using similar argument 



with /o replaced by Zq + 1, ^i + 1 by /i. 

(ii) : We can follow the argument of the first conclusion line by line 
except with the set Sa,b,c replaced by its subset Va^^c- We omit the 
detailed arguments here. D 



Proof of Proposition 3^. Suppose on the contrary that l\/\.a,b,c(t)^o = 
Ma^(,^c(^)xi = 1 for two distinct vectors xo,xi G B^. Then there exists 
7^ Aq G 6Z such that xo(Ao) 7^ xi(Ao) while xo(A) = xi(A) for all 
|A| < |Ao|. Without loss of generality, we assume that xo(Ao) = 1 and 
xi(Ao) = 0. Let Ai G bZ be so chosen that AiAo > 0, |Ai| < |Ao| 



and Ai — A 



o| — mmAg(-|Ao|,|Ao|)nM,xo(A)=xi(A)=l 



|A- Ao|. Thus Ma,b,c{t- 
Ai)rAiXo = Ma,b,c{^ - Ai)rAiXi = 1 by (Q, and both ta^xo and ta^xi 
belong to i3° by the selection of the index Ai. As Aq — Ai is the clos- 
est positive (or negative) index to zero such that rAiXo(Ao — Ai) = 1. 
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Then it follows from Proposition 3.6 that rA^xi(Ao 
contradicts to xi(Ao) = 0. 



Ai: 



1, which 

D 



Proof of Proposition \3.1(^ Suppose, on the contrary, that (3.54) does 
not hold. Then there exists a nonnegative integer m such that (-Ra,6,c)™([c- 
Co,c - Co + b - a) + oL) n Sa,b,c 7^ and {Ra,b,cT{[c - Cq, c - Cq + 
6 — a) + aZ) r\ Sabc = for all < n < -m. We observe that 



m ^ (or equivalently m is a positive integer) by Proposition 3.6 
Take t G (i?a,6,c)™'([c — cq, c — cq + 6 — a) + aZ) fl Sa,b,c- Then t = 
Ra,b,c{^) for some s G (-Ra,6,c)'"^^([c — Co, c — Co + 6 — a) + aZ). If 
s G [cq + a — &, Co) + aZ, then t = s by (2.13), which is a contra- 
diction as ([co + a — b, Cq) + aZ) fl Sa^b, 

s ^ [co + a - 6, Co 

sition 3.8, Hence s G {Ra,b,c)"^~^ {[c — cq,c — cq + b — a) + aZ) fl Sa,b,c, 
which is a contradiction. D 



by Proposition 3.6 If 



aZ, then s = Ra,b,ci^) G <Sa,b,c by (3.50) and Propo- 



4. Maximal Invariant Sets of Piecewise Linear 
Transformations 



In this section, we prove Theorem |2.3[ To do so, we need the most 
crucial observation of this paper that Sa,b,c is the maximal set such that 



Ra,b,c<Sa,b,c C O^ 



and Ra,b,c<Sa,b,c C Safi^ci 



(4.1) 

and 

(4.2) 

Sa,b,c n ([c - Co, c - Co + & - a) + aZ) = Sa,b,c n ([co + a - 6, cq) + aZ) 



c.f. Lemma 5.11 for the case that a/b G 



Theorem 4.1. Let {a,b,c) be a triple of positive numbers satisfying 



a <b < c andb — a < co := c— [c/b\b < a, the set Sg^bx b e as in 1 2.9), 
and let transformations Ra,b,c and Ra,b,c be as in (2.13) and (2.14). 
Then 

(i) Safi,c is the maximal set that is invariant under the piecewise 
linear transformations Ra,b,c o,nd Ra,b,c, o,nd that has empty in- 
tersection with their black holes. 
(ii) The complement of Sa,b,c is the minimal set that is invariant 
under the piecewise linear transformations Ra,b,c o,nd Ra.b,c, o,nd 
that contains their black holes. 



To prove Theorem 2.3 (and the characterization of (2.11 ) in Theorem 



5.7), we need the following deep connection between the invariant sets 

T^a,b,c and Sa,b,c- 
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Theorem 4.2. Let (a, b, c) be a triple of positive numbers with a < b < 
c and b — a < Co := c — [c/b\b < a, and let the sets Vafi,c and Sa,b,c be 
as in p^ and (K^ ). Then 

Va,b,c = {Sa,b,c n ([0, Co + a - 6) + aZ) n iSa,b,c - [c/b\b)) 

(4.3) U(5a,fe,c n (lJ\e[b,{[c/bi-l)b]nbz{<Sa,b,c — A))). 

In next two subsections, we prove Theorems |4.1 and 42, and apply 



them to prove Theorem 2.3 



4.1. Maximal invariant sets. To establish maximality in Theorem 



4.1, we characterize whether a particular point belongs to the set Sa^^c 



which will also be used in the proofs of Theorems 5.4 and|5.5 



Proposition 4.3. Let {a,b,c) be a triple of positive numbers satisfying 
a < b < c and b — a < Cq := c — [c/b\b < a, and let Ra,b,c and Ra,b,c 
be as in ( 2.1!^ and (2.14)- Then t E Sa,b,c if and only if {Ra,b,cY{t) 
and {Ra,b,c)"'it),n > 0, do not belong to the black holes of the piecewise 
linear transformations Ra,b,c and Ra,b,c- 



Now assuming that Proposition 4.3 holds, we start to prove Theorem 



an 



Proof of Theorem ^.i, (i) By (3.30) in Proposition 3.6, and (3.51) in 



Proposition 3.8, the set Sa,b,c is an invariant set under piecewise lin- 
ear transformations Ra,b,c and Ra,b,c which has empty intersection with 
their black holes. Then it suffices to show the maximality. Let E be 
an invariant set under piecewise linear transformations Ra,b,c and Ra,b,c 
that has empty intersection with their black holes. Take t E E. Then 
it follows from the invariance of the set E that 



(4.4) 



{Ra,b,cT{t) e E and {Ra^b,cT{t) e E 



for all nonnegative integers n. This, together with Proposition |4.3| and 
the empty intersection assumption between the set E and the black 
holes of the piecewise linear transformations Ra,b,c and Ra,b,c, implies 
that t G Sa,b,c- Thus E C Sa,b,c and the maximality of the set Sa,b,c 
follows. 

(ii) By (3.30) in Proposition 3.6 and (3.51) in Proposition 3.8, the 



complement of Sa,b,c is an invariant set under the piecewise linear trans- 
formations Ra,b,c and Ra,b,c, and it contains their black holes. For any 
set E that is invariant under the transformations Ra,b,c and Ra,b,c that 
contains their black holes, we obtain from (3.50 ) that Rafi^dt), Ra,b,c(t) ^ 



E for any t ^ E. This, together with Proposition 4.3 , proves the desired 
minimality for the complement of Sa^^c- D 
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Next we prove Proposition |4.3[ 



Proof of Proposition 4-3 (^^) Take t G Sa,b,c- Then (-R a,b,c) "(t) G 



Safi^c and (-Ra,6,c)"(^) £ '^a^b^c for all n > by Proposition 3.8 This 
together with Proposition 3.6 proves the desired empty intersection 
property with black holes. 

{•^=) Take any real number t such that {Ra,b,c)"'(t) and {Ra,b,c)"' (t) , n > 
0, do not belong to the black holes of the transformations Ra,b,c and 
Ra,b,c- Define 

r (RaAcTit) if ^ > 1 
t^= } t ^ if ra = 

i (^a,6,c)-"(t) if n < 0, 

and Xn = tn — t,n & Tj. Then 

(4.5) tn+m = {Ra,b,cT{tn) for all n G Z and < m G Z 
and 

(4.6) A„ G 6Z and A„+i - A„ G { [c/6j 6 + 6, [c/h\ h} for all n G Z, 



by (3.50), and the assumption that t„ ^ [cq + a — 6, Cq) + oL for all 
n > and tn ^ [c — Co, c — Co + 6 — a) + aZ for all n < 0. Define 
xt(A) = 1 if A = Art for some n G Z and Xi(A) = otherwise, and set 
xt = (xt(A))AG6z- Then xj belong to B^. Let ;U„ G aZ be so chosen that 
in '■= tn — fJ'n ^ [0,a). Then {fin}n& is a strictly increasing sequence 
with 

(4.7) lim fin = +00 and lim /i„ = — oo 

n—^+OD n—>—oo 



by (4.6), and 

^ X[0,c)it - /in + A)Xi(A) = ^ X[0,c)(i - fin + K 



AefeZ 



(4.8) = ^ X[0,c)itn^ - Hn) = X[0,c)itn - /in) = 1 for all 77, G Z, 

mGZ 

where the first equation follows from the definition of the vector x^ and 
the third one holds as t^ ~ /in < ^n ~ /^n ~ ^ < for all m < ri and 

tm- IJ'n> {tn+1 " ^n) + {tn " jJ'n) = (A„+l - An) + (t„ - jJn) > C for all 

m > n. Similarly for any /i G aZ with fin < fJ' < /in+i, 

(4-9) ^ X[o,c)(t - /i + A)xt(A) = Y^ X[o,c)(im - /i) = 1 

AefeZ mez 

as tm — /i < t„ — /i < /i„ + a — /i < for m < 72, < t„+i — /in+i < 

tm~l^ ^ tn+l~l^n~Ci < C for 771 = TZ+l, andtm~/i > i^n+2~/in+l + > C 
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for m > n + 2. Combining (4.8) and (4.9) proves lSA.a^b,c{t)^t 
hence t E Sabc- 



1 and 

D 



We conclude this subsection by the proof of Theorem 4.2 



Proof of Theorem \4-^ We use the double inclusion method. Take t G 
Vafi^c- Let X e i3° be so chosen that Mafec(^)x = 2 and denote by K 
the support of the vector x; i.e., the set of all \ E IfL with x(A) = 1. 
Write K = {Xj : j E Z} for some strictly increasing sequence {Xj}'^_^ 
in bZ with Aq = 0, and define x, := (xj(A))AeaZ, ^ = 0, 1, by Xj(A) = 1 if 
X E Ki and Xj(A) = otherwise, where Ki = {Aj+2j : j E Z}, i = 0,1. 
Then following the argument in the proof of the implication (ii)^=^(iii) 
of Theorem 3^, we have that 

(4.10) xo, TA.Xi E Bl and M,,b,,(t)xo = M„,fe,,(t + Ai)rA,xi = 1. 

Notice that T)a^hfi fl ([cq + a — 6, cq) + aZ) = by the supset property 
T^afifi C Safi^c in (3.27) and the trivial intersection property 5a_(,,cn([co + 
a — b, Co) + aZ) = in (3.30). Then either t E [0, cq + a — 6) + aZ or 
[co,a) + aZ. For the first case that t G [0,Co + a — 6)+aZ, A2 = [c/b\b+b 
by ( [3!28| ) and ( |33I| in Proposition [3^ Hence Ai E [b, [c/b\b]nbZ and 
t + XiESa,b,chi^A^. Thus 

(4.11) t G {Sa,b,c^{[0,Co + a-b) + aZ)) n {Ux^elb,lc/b\b]nbz{Sa,b,c- Xi)) 

for the first case. Similarly for the second case that t E [cq + a — 6, a) + 
aZ, X2 = [c/b\b by (3.28) and (3.31) in Proposition 3.6, which together 



with (4.10) implies that 

(4.12) t E {Sa,b,c n ([Co, a) + aZ)) n ( Ua, 



e[fe,(Lc/bJ-i)6]nra 



{Sa,b,c — -^1)) 



for the second case. Combining (4.11) and (4.12) proves the first in- 
clusion 



V, 



a,b,c 



(4.13) 



C {Sa,b,c n ([0, Co + a - 6) + aZ) n (5„,5,c - [c/b\b)) 

^{<Sa,b,c n i^Xe[b,{lc/b\-l)b]nbzi<Sa,b,c ~ -^)))- 



Take t E Sa,b,c H ([0, Co + a — &) + aZ) fl {Safi,c — [c/b\b). Then there 
exist Xo,Xi G B^ such that 



(4.14) 



M,,fe,e(t)xo = M,,b,,(t + Lc/6Jfe)xi = 1. 



Define x = xq + T^y(,/b\b^i- By (3.1) and (4.14), we have that 

(4.15) M„,fc,e(t)x = M,,b,,(t)xo + M,,b,,(t + [c/b\ 6)xi = 2. 

Now let us verify that x G B^. Write x = (x(A))Ae6z- Observe that 
x(A) G {0, 1,2} for all A G 6Z and x(0) > xo(0) > 1. Then it suffices 
to prove that x(A) 7^ 2 for all A G bZ. Suppose, to the contrary, that 
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x(Ao) = 2 for some Aq G hTL. Then xo(Ao) = 1 and r_Lc/fej6Xi(Ao) = 1. 
Hence TaoXq, rAo_Lc/6jfeXi G El and 

Ma,fe,c(t + Ao)rAoXo = Ma,6,c(t)xo = 1 and y\-a,hA^ ^ \Y\a~\clh\hy^\ = 1 

by (3.1) and (4.14). Thus taqXo = Txg-ic/b\b'^i by Proposition 3.9, which 
is a contradiction as T_\^c/b\b'^i{[c/b\b) = xi(0) = 1 by the assumption 
that xi G B^ and xo([c/6J6) = by (3.31) and the assumption that 
t G [0, Co + 6 — a) n iSa^fe^c- Therefore x is a binary vector in B^. This 
together with (4.15) proves that 

(4.16) Sa,b,c n ([0, Co + a - 6) + aZ) n {Sa,b,c - [c/b\ b) C Va,b,c. 
Applying similar argument, we can prove that 

(4.17) Sa,b,c n ([0, Co + a - 6) + aZ) f] {Sa,b,c - A) C Va,b,c 
for all A G [b,{[c/b\ - 1)6] n 6Z, and 

(4.18) Sa,b,c n ([co, a) + aZ) n (5„,,,, - A) C Va,b,c 



for all A G [6, ([c/6J — 1)6] n6Z. The desired equality (4.3) then follows 
from ( |4l3| , ( |4?16| ), ( [4171 ) and ( |4l8| ). D 



4.2. Proof of Theorem 2.3 The main ideas behind the proof of 
Theorem |2.3 are as follows. The conclusion (VIII) follows from the 



results in ^l Section 3.3.3.5, 3.3.3.6 and 3.3.4.3]. We include a different 
proof by showing that the only binary vector x satisfying M^ f„c(^)x = 2 
is the vector 1. We prove Conclusion (IX) by showing that any point 
not in the black hole [cq + a — 6, Co) + aZ of the transformation Rafi,c 
is contained in the hole {Ra,b,c)'^[c — co, c — co + 6 — a) + aZ for some 
n > 1, see (4.21). The crucial step in the proof of Conclusion (X) (resp. 
Conclusion (XI)) is to show that Sa,b,c = [0, co + a — 6) + aZ in (4.24) 
(resp. Sa,b,c = [co, a) + aZ in ( |4.25[ )). 

Proof of Theorem 2.3[ (VIII): Suppose on the contrary that Q{x[o,c), oZx 
Z/b) is not a Gabor frame. Then by Theorem 3.2 there exist t G M 
and (x(A))Ae6Z G B^ such that 



(4.19) 



y X[o,c)it — /i + A)x(A) = 2 for all fi G aZ. 



xebz 



By the assumption [c/6J = 1 and b < c, given any t G M and /i G aZ, 
the equality X[o,c)(^ — /i + A) = 1 holds for at most two distinct A G bZ. 
This together with ( |4.19| ) that x(A) = 1 for all A G bZ, and also that 

(4.20) t- ij^[c, 26) + 6Z for all /i G aZ. 

If a/6 ^ Q, then there exists /xo G aZ by the density of the set aZ+6Z 
in M such that t — Hq E [c, 26) + 6Z, which contradicts to (4.20). 
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If ajh G Q, then a/h = p/q for some positive coprime integers p and 
q. Hence 

t ^ [c, 26) + bZ/q = R, 



where the first conclusion follows from (4.20) and the equality holds as 
2b — c = b — Co>b — a> b/q by the assumption 0<b — a<Co<a. 
This is a contradiction. 

(IX): By the supset property (3.27) and Theorem 3.2, it suffices to 
prove the following result. 

Proposition 4.4. Let {a,b,c) be a triple of positive numbers that sat- 
isfies a < b < c,b — a < Co := c — [c/b\b < a, [c/b\ > 2, Ci = 
c — Co — [(c — Co)/aJa > 2a — b, and let Sa^b,c be as in (2.9). Then 

Sa,b,c = 0. 

Proof. By Propositions |3.6 and 3.10[ it suffices to prove 

(4.21) [co-a,Co+a-6)+aZ C U^=o(^aAc)"([c-Co, c-Co+6-a)+aZ), 

where L = max(L(co + a — 6)/(ci + b — 2a)\, [(a — Co)/(a — Ci)J). 

For any t E [0,Co + a — b), write t = /(ci + b — 2a) + t' for some 
t' G [0, min(ci + 6 — 2a, cq + a — b)) and < I < L. Then 

t e {RaMYit') + aZc {RaAcYilO, ci + b-2a) + aZ) 

(4.22) C uLo(^aAc)"([c-co,c-co + 6-a) + aZ) 

for all t G [0,Co + a — b). 

Similarly for any s G [cq — a,0), let s = /'(ci — a) + s' for some 
s' G [max(ci — a, Cq — a), 0) and < I' < L. Then 

s G {RaAcY'is) + aZc {RaAcY' {[ci - a, 0) + aZ) 

(4.23) C U^=o(^aAc)"([c-Co,c-Co + 6-a) + aZ) 



for all s G [ci — a,0). Combining (4.22) and (4.23) and applying the 
periodic property (3.3) proves (4.21). D 



(X): Mimicking the argument used to prove (4.23), we can show 
that 

U^=o(/2a,6,c)"([c - co,c- co + b-a) + aZ) = [co + a-b,a) + aZ, 



which implies that the set Sa,b,c in (2.9) is given by 

(4.24) Sa,b,c = [0, Co + a - 6) + aZ. 

From the assumption on Ci, the ratio a/b is rational. We write a/b = 
p/q for some coprime integers p and q. Clearly p > 2 as 6 — a < Cq < a. 
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By the assumption that Ci = 2a — b, we have that [c/b\ + 1 G pZ, 
which imphes that 

Ra,b,c(t) —tEoL for all t G Sa,b,c = [0, Co + a — &) + oL. 



This together with Theorem |3. 2 and Proposition 4.2 implies that the 



Gabor system ^(x[o,c); c^Z x Z/6) is a frame of L if and only if P, 



a,b,c 



in (2.2) is an empty set if and only if ([0, cq + a — 6) + aZ) fl ([0, cq + 
a - 6) + A + aZ) = for all A G [6, [c/h\h] n 6Z. Observe that ([0, cq + 
a - 6) + aZ) n ([0, Co + a - 6) + A + aZ) = [0, co + a - 6) + aZ ^ 
for X = pb E [b, [c/b\b] fl 6Z provided that [c/6J > p, and also that 
([0,co + a-6)+aZ)n([0,co + a-6) + A + aZ) = [b/q,co + a-b)+aZ ^ 
for X = kb E [b, \_c/b\ b] fl 6Z where l</i;<]9 — lis the unique integer 
such that qk~l E pZ, provided that [c/b\ +1 = p and Cq + a — 6 > b/q. 
Therefore the assumptions that \_c/b\ +1 = p and Cq + a — 6 < b/q are 
necessary for the Gabor system Q{x[o,c), aZ x Z/6) being a frame of L^. 
On the other hand, if [_c/b\ + 1 = p and co + a — b < b/q, one may verify 
that ([0, Co + a - 6) + aZ) n ([0, Co + a - 6) + A + aZ) = ([0, co + a - 6) + 
aZ) n ([0, Co + a - 6) + k{X)b/q + aZ) = for all A G [1, Lc/&J]& n 6Z, 
where k{X) is the unique integer in [l,p— 1] such that k{X)b/q — X G aZ. 
Therefore the assumptions that \_c/b\ + 1 = p and co + a — 6 < b/q is 
also sufficient for the Gabor system Q{x[oc),o-'Z x Z/6) being a frame 
of L\ 



(XI) Mimicking the argument used to prove (4.22), we may show 
that 

(4.25) Sa,b,c = [co, a) + aZ. 

Now we can apply similar argument used in the proof of the conclusion 
(X) of this theorem. From the assumption that ci = 0, it follows 
a/b = p/q for some coprime integers p and q with p >2 and [c/6J G pZ. 
By (4.25) and Theorem 3.2, we can show that Q{x[o,c),0''Z x Z/6) is a 



frame of L^ if and only if ([co, a) + aZ) fl ([cq, a) + A + aZ) = for all 
A G [6, [c/6J b — b] n 6Z. Then the desired necessary condition for the 
Gabor system Q{x[o,c), c-Z x Z/6) being a frame of L^ follows from the 
observation that ([co, a) + aZ) fl ([cq, a) + pb + aZ) = [co, a) + aZ 7^ 
if [c/b\ > p + 1, and that ([cq, a) + aZ) fl ([co, a) + kb + aZ) = [cq, a — 
b/q) + aZ 7^ if \c/b\ = p and a — Co > b/q where 1 < k < p — 1 
is the unique integer such that qk + 1 E pZ. The sufficiency for the 
conditions that [c/b\ = p and a — Cq < b/q holds as ([co,a) + aZ) fl 
([co, a) + A + aZ) = ([co, a) + aZ) n ([co, a) - k{X)b/q + aZ) = dS for all 
A G [1, [c/6J]6 n bZ, where A;(A) is the unique integer in [l,p — 1] such 
that k{X)b/q + X E aZ. D 
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5. Properties of Maximal Invariant Sets 



To prove Theorems 2.4 and 2.5, we need some deep properties of 



the maximal invariant sets Sa,b,c- Let us start from some examples of 
maximal invariant sets Sa,b,c- 

Example 5.1. Take a = 7r/4, 6 = 1 and c = 23 - ll7r/2. The black 
holes of the corresponding transformations Ra,b,c and Ra,b,c are [17 — 

2l7r/4, 18 - 1 l7r/2) + 7rZ/4 and [5 - 37r/2, 6 - 7tt'/A) + 7rZ/4 'respectively, 
which can be transformed back and forth via the middle hole [11 — 

77r/2, 12 - 157r/4) + 7rZ/4; i.e., 

Ra,b,ci[^ - 37r/2, 6 - 77r/4) + 7rZ/4) = [11 - 77r/2, 12 - 157r/4) + 7rZ/4 
{Ra',b,c?{[5 - 37r/2, 6 - 77r/4) + ttZ/A) = [17 - 2l7r/4, 18 - ll7r/2) + 7rZ/4. 

So the maximal invariant set 

SaAc = [18 - 237r/4, 11 - 77r/2) U [12 - 157r/4, 5 - 37r/2) 
U[6 - 77r/4, 17 - 2l7r/4) + 7rZ/4 
^ [-0.0642, 0.0044) U [0.2190, 0.2876) U [0.5022, 0.5066) + 0.7864Z 

consists of intervals of different lengths on one period and contains a 
small neighborhood of the lattice 7rZ/4, c.f. Figure |2J 



Example 5.2. For the triple {a,b,c) = (13/17,1,77/17), the black 
holes of the corresponding transformations Ra,b,c and Ra,b,c are [5/17, 9/17) + 
13Z/17 and [3/17, 7/17) + 13Z/17. Applying the transformation Ra,b,c 
to the black hole of the transformation Rabci we obtain that 
(5.1) 

i?aAc([3, 7)/17 + 13Z/17) = ([5, 7) U [10, 12))/17 + 13Z/17, 
{Ra,b,cf{[^, 7)/17 + 13Z/17) = ([5, 7) U [0, 2))/17 + 13Z/17, 
{Ra,b,c)\[^, 7)/17 + 13Z/17) = [5, 9)/17 + 13Z/17. 

Thus the maximal invariant set 

Sa,b,c = ([2,3)U[9,10)U[12,13))/17+13Z/17 

^ [0.1176, 0.1764) U [0.5294, 0.5882) U [0.7059, 0.7647) + 0.7647Z 

consists of intervals of same length 1/17 on the period [0, 13/17) and 
contains small left neighborhood of the lattice 13Z/17, and its comple- 
ment R\Sa,b,c = ([0, 2) U [3, 9) U [10, 12))/17 + 13Z/17 contains one big 
gap of size 6/17, two small gaps of size 2/17 on the period [0, 13/17), 
and a small gap attached to the right-hand side of the lattice 13Z/17. 
For the triple (a, b, c) = (13/17, 1, 73/17), the maximal invariant set 
\b,c = ([0,1) U [7,8) U [10,11))/17 + 13Z/17 contains a small right 
neighborhood of the lattice 13Z/17, while its complement M\Safi^c = 
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([1, 7) U [8, 10) U [11, 13))/17 + 13Z/17 contains a small gap attached 
to the left-hand side of the lattice 13/17. 

For the triple (a, b, c) = (13/17, 1, 75/17), the maximal invariant set 

Sa,b,c = ([0,3)U[7,10)U[10,13))/17 + 13Z/17 

= [0, 0.1765) U [0.4118, 0.5882) U [0.5882, 0.7647) + 0.7647Z 

consists of intervals of "same" length 3/17 and contains small left and 
right neighborhoods of the lattice 13Z/17. On the other hand, its 
complement M\Safi^c = [3, 7)/17 + 13Z/17 contains one big gap of size 
4/17 and two small gaps of size "zero" at {0, 10/17} on the period 
[0,13/17), c.f. Figure |2l 

Example 5.3. For the triple {a,b,c) = (6/7,1,23/7), black holes of 
the corresponding transformations Ra,b,c and Ra,b,c are [1, 2)/7 + 6Z/7 
and [3, 4)/7 + 6Z/7 respectively. Observe that 

Ra,bA[^, 4)/7 + 6Z/7) = [0, l)/7 + 6Z/7 
{RaAcfil^, 4)/7 + 6Z/7) = [4, 5)/7 + 6Z/7 
{Ra,b,cf{[3, 4)/7 + 6Z/7) = [1, 2)/7 + 6Z/7, 

which also implies that i?a,fe,c([3, 5)/7 + 6Z/7) = [0, 2)/7 + 6Z/7. There- 
fore the maximal invariant set 

Sa,b,c = [2,3)/7U[5,6)/7 + 6Z/7 

= [0.2857, 0.4286) U [0.7143, 0.8571) + 0.8571Z 

consists of intervals of length 1/7, while its complement M\Sa,b,c = 
([0, 2) U [3, 5))/7 + 6Z/7 consists of gaps of length 2/7, c.f. Figure |2| 

In the above examples, we see that the black hole [co+a—b, co)+aZ of 
the transformation Ra,b,c attracts the black hole [c— cq, c— co-|-6— a)+aZ 
of the other transformation Ra,b,c when applying TZa,b,c finitely many 
times, i.e., 

(-Ra,b,c)^([c - Co, C - Cq + b - a) + oL) = [cq + tt - b, Co) + aZ 

for some nonnegative integer L, and that holes 

(-Ra,6,c)'([c — Co, c — co + 6 — a) +aZ) = [co + a — &, co) + aZ, < I < L — 1, 

may or may not overlap with the black hole [co + a — b, Cq) + aZ of the 
transformation Ra,b,c, which depends on the ratio a/b being irrational 
or not. 

For the first case that the ratio between time-spacing parameter a 
and frequency-spacing parameter b is irrational (i.e. a/b ^ Q), we show 
that if Sa^b,c 7^ then the black hole [co + a — b, Cq) + aZ of the trans- 
formation Ra,b,c and the black hole [c — Co, c — Co + & — a) + aZ of the 
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transformation Ra,h,c are inter-transformable through mutually disjoint 
periodic holes (-Ra,fe,c)"([c - Cq, c - Cq + 6 - a) + aZ) = (-Ra,fe,c)^~"([co + 
a — b, Co) + aZ), < n < D, where D is a nonnegative integer. Further- 
more the complement of the set Sa,b,c is the union of mutually disjoint 
holes of same size, but the set Sa,b,c is the union of disjoint intervals of 
"different" sizes. 

Theorem 5.4. Let (a, b, c) be a triple of positive numbers satisfying 
a < b < c,b - a < Co := c - [c/b\b < a, [c/b\ > 2 and a/b ^ Q. 
Assume that Sa.h,c 7^ 0- Then there exists a nonnegative integer D < 
[a/(6 — a)\ — 1 such that 

(5.2) (i?aAc)"(co + a - 6) + aZ = (/?a,v)''""(c - cq) + aZ 

for allO<n<D, 

(5.3) 

((^aAc)"(c-Co) + [0,6-a] + aZ)n((i?,,fe,J"'(c-Co) + [0,6-a] + aZ) =0 

for all < n y^ n' < D . Moreover 

R\Sa,b,c = U^^o((^aAc)"(c-Co) + [0,6-a) + aZ) 

= U^=o(^a,fe,c)"([c - Co, c - Co + 6 - a) + aZ) 

= U^=o(^a,b,c)"([co + a - 6, Co) + aZ) 

(5.4) = u^=o((^aAc)"(co + a-&) + [0,6-a) + aZ). 

For the second case that a/b is rational, we write a/b = p/q for some 
coprime integers p and q. We restrict ourselves to consider c G bZ/q 
because for c bZ/q, Q{xio,c), dZ x Z/6) is a Gabor frame if and only if 
both G{x[o,iqc/b\b/q), oL X Z/6) and ^(x[o,LW6+iJ V<?)' "^ ^ ^/^) ^^^ Gabor 
frames, see [251 Section 3.3.6.1] and the conclusion (XIV) of Theorem 



2.5, Observe that for a/b = p/q and c G 6Z/g, 



(5.5) Ma,6,e(t) = Ma,fe,,([gt/6J6/g), t e M, 
which implies that 

(5.6) Va,b,c = Va,b,cf^bZ/q+[Q,b/q) and Sa,b,c = Sa,b,cf^bZ/q+[Q,b/q). 

Even further, the sets Va,b,c and 5a,fe,c are essentially finite sets, as 
they are completely determined by their restrictions to the finite set 

{o,6/g,...,(p-i)&M, 

, . { Va,b,c = Va,b,c n {0, b/q, ...,ip- l)b/q} + pbZ/q + [0, b/q) 

^ ' 1 5,Ac = 5a,,,,n{o,Vg,---,(p-i)&M+p&z/g + [o,V9), 
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by (5.6) and the periodic property and (3.3). In the next theorem, 
we show that the maximal invariant set Sa,b,c is the union of half- 
open intervals of same size and its complement M\Sa,b,c is the union of 
mutually disjoints gaps of "two different" sizes. 

Theorem 5.5. Let {a,b,c) be a triple of positive numbers satisfying 
a<b<c,b — a<Co := c — [c/b\b < a, [c/b\ > 2, a/6 = p/q for 
some coprime integers p and q, and c/b G Z/q. Assume that Sa,b,c 7^ 0- 
Then there are 5 G [0, cq + a — 6] fl 6Z/g, 5' G [cq — a, 0] fl 6Z/g, and 
nonnegative integers Ni and N2 with the following properties: 

(i) At least one of 6 and 6' is equal to zero; i.e., 

(5.8) 66' = 0. 

(ii) The periodic gaps {Ra,b,c)"'{c—Co+[6', b—a+6))+aZ, < n < Ni, 
have length b — a + 6 — 6', and the periodic gap {Ra,b,c)'^^{c~Co + 
[6',b — a + 6)) + aZ coincides with [co + a — b — 6,Co — 6') + aZ that 
contains the black hole of the piecewise linear transformation 

^a,b,C! I'.C., 

{Ra,b,cT{c-Co + [6',b-a + 6))+aZ 

(5.9) = {Ra,b,cT{c-co + b-a + 6) + [a-b + 6' -6,0) + aZ 

for all < n < Ni, and 

(5.10) {Ra,b,c)^' {c-CQ + b-a + 6) + aZ = co-6' + aZ. 

(iii) The periodic gaps {Ra,b,c)"^ {[cq + a — b — 6,Co — 6')\[cq + a — 
b, Co)) + aZ, 1 < m < N2, have length 6 — 6' and the periodic 
gap (-Ra,6,c)"'([co + a-b- 6,cq- 6')\[cq + a - b, cq)) + aZ with 
m = N2 is the same as [6', 6) + aZ, provided that 6 — 6' ^ 0; 
i.e., 

(-RaAc)''"([co + a-b-6,co- 6')\[co + a - b, cq)) + aZ 

(5.11) = {Ra,b,cr{co-6') + [6'-6,0) + aZ 

for all 1 < m < N2, and 

(5.12) {Ra,b,cf'{[co+a-b-6,Co-6')\[co+a~b,co))+aZ= [6',6)+aZ. 

(iii)' (-Ra,6,c)^^(co) G aZ provided that 6 = 6' = 0. 

(iv) The periodic gaps {Ra,b,c)"'{c—co+[6', b—a+6))+aZ, < n < Ni, 
of length b — a + 6 — 6', and {Ra^b,c)"^{[co + ci — b — 6, cq — 6')\[co + 
a — 6, Co)) + aZ, 1 < m < N2, of length 6 — 6' together with 
neighboring intervals of length b/{2q) at each side are mutually 
disjoint, provided that 6 — 6' j^ 0. 



46 XIN-RONG DAI AND QIYU SUN 

(iv)' The periodic gaps {Ra,b,c)^{\c — cq, c — cq + 6 — a)) + aZ, < 
n < Ni, of length h — a associated with neighboring intervals of 
lengthb/{2q) at each side, and {Ra,b,c)"'{co) + [-b/{2q),b/{2q)) + 
aZ, I < m < N2, are mutually disjoint, provided that S = 5' = 
0. 
(v) The complement of the set Sa,b,c is the union of periodic gaps 
{Ra,b,cT{c-co+[S',b-a+6))+aZ,0< n < Ni, and {Ra,b,cr{[co+ 
a — b — 5,Cq — 5')) + aZ, 1 <m < N2; i.e., 

M\5a,v = {y^nU{Ra,b,cT{c-c^ + [6\b-a + 6)) + a'L) 
U( ulU {RaAcTlco + a-b-5,Co-5') + aZ) 

(5.13) = U^i^'''{Ra,b,cr{c-Co + [6',b-a + 6) + aZ). 

(vi) The set Sa,b,c is composed of intervals of same length, 

(5.14) Sa,b,c = U^i+^^(/?,,fe,e)"(c -co + b-a + 6) + [0,h) + aZ 
where 

(5.15) {Ni + N2 + l){h + 6-6') + {Ni + 1)(6 - a) = a. 

We remark that for triples (13/17, 1, 77/17), (13/17, 1, 73/17) and 



(13/17, 1, 75/77) in Example 5.2 and (6/7, 1, 23/7) in Example 5.3, the 



corresponding pair {6, 6') in Theorem 53]is given by (2/17, 0), (0, —2/17), 
(0,0), (1/7,0) respectively. 

Listed below are some comparisons of the maximal invariant set Sa,b,c 
for the first case that a/b ^ Q and for the second case that a/b = p/q 
for some coprime integers p and q and c G bZ/q, where 6, 6' are as in 
Theorem 5^, c.f. Examples 5A_^ 5^ and 5.3[ and Figure [2] 



For the second case, the restriction of the maximal invariant 
periodic set Sa,b,c on the period [0,a) is the union of finitely 
many half-open intervals of same size ii 6 — 6' > 0, while for 
the first case it contains the union of finitely many half-open 
intervals of different sizes. 

For the second case, the restrictions of the periodic complement 
^\<Sa,b,c on the period [0, a) consists of finitely many mutually 
disjoint periodic gaps of lengths b — a + 6 — 6' and 6 — 6' ii 
6 — 6' > 0, while for the first case it includes finitely many 
mutually disjoint periodic holes of same size b — a. 
For the second case, either [0, 6) or [6', 0) is a gap contained in 
the periodic complement M\Sa,b,c ii 6 — 6' > 0, while for the 
first case no left or right holes at the origin are contained in the 



periodic complement M.\Safi,c, see Lemma 3.11 
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The complement of the maximal invariant set Sa,b,c for the sec- 
ond case with 6 = 6' = looks "similar" to the one for the 
first case, since their restrictions on one period both consist 
of finitely many mutually disjoint half-open intervals of length 
b — a. On the other hand, they are "different" if we adhere inter- 
vals of sufficiently small length e to each side of those half-open 
intervals. For the second case, those gaps with neighboring 
intervals has the following "loop" structure via the piecewise 
linear transformation Ra,b,c with shrinking and expanding at its 
black hole and the origin: 



Co + [— e, e + 6 — a) + aZ 



i 



I 



Co + a 



b,Co) 



+ aZ 





[— e,e) + oL 






t 
t 




Ra,b,c{co) + [-e, e) + aZ 



Hence the piecewise linear transformation Ra,b,c has "finite" or- 
der, as there exists a positive integer L such that {Ra,b,c)^ is an 
"identity" operator / in the sense that {Ra,b,c)^ (t) + a'^ = t + oL 
for all t G Sa,b,c, see Theorem 5.8 But for the first case, ap- 
plying the transform Rafi,c finitely many time, the black hole of 
the piecewise linear transformation Ra,b,c is "totally" attracted 
by the black hole of the piecewise linear transformation Ra,b,c, 
which indicates that the piecewise linear transformation Ra,b,c 
has "infinite" order. 

The complement of the maximal invariant set Sa^^c for the case 
that N2 = 0, c.f. Example 5.3, looks also "similar" to the one 
for the first case, in the sense that their restrictions on one 
period both consist of finitely many mutually disjoint half-open 
intervals of same length, but the lengths are different for those 
two cases. Also for the case that A''2 = 0, the maximal invariant 
set Sa,b,c is the union of finitely many half-open intervals of same 
size. 

For both cases, the black hole of the piecewise linear trans- 
formation Ra,b,c attracts the black hole of the piecewise linear 
transformation Ra,b,c when applying the piecewise linear trans- 
formation Rabc finitely many times. 



[ci, Ci + b — a) + aZ 



[co + a — b, Co) + aZ 
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see (5.36|), (|5.61|) and (|5.68|). Moreover 



b,c 



R\{ U^=o {RaAcnici, ci + b-a) + aZ)) 
(5.16) = M\(u^=o(^a,v)"([ci,Ci + 6-a) + aZ)) 

for some nonnegative integer L > when Sa,b,c 7^ 0- 

Having explicit construction of the maximal invariant set Sa^b,c in 



Theorems |5.4| and |5.5[ we next consider covering property of the max- 
imal invariant set Sa,b,c and then its application to characterize (2.11). 



Theorem 5.6. Let (a, b, c) be a triple of positive numbers satisfying 
a < b < c, [c/b\ > 2,b — a < Co := c — [c/b\b < a and < Ci : = 
c — cq— [{c — co)/a\a < 2a — b, and let Sa,b,c be as in (2.9). If Sa,b,c 7^ 
and either a/b ^ Q or a/b = p/q and c G bZ/q for some coprime 
integers p and q, then 

(5.17) (5,,6,en([0,Co+a-&)+aZ)+[c/6J6)u(ut^;J-'(5,,f,,,+A;6)) = M. 

As a direct application of the above theorem, we have that 

^t=t\Sa,b,c + A;6) C M C dSo\Sa,b,c + kb)- 

So roughly speaking, the maximal invariant periodic set Sa,b,c is either 
an empty set or its ( \c/b\ + 1) copies would cover the whole line. Recall 
that the set 'Dabc can be obtained from the maximal invariant set Sabc 



by some set operations (Theorem 4.2). This together with Theorem 5.6 



leads to the following equivalence between the empty set property for 
Da fee and an equality about the Lebesgue measure of the maximal 
invariant set Sa,h,c- 

Theorem 5.7. Let (a, 6, c) be a triple of positive numbers satisfying a < 
b < c,b- a < Co:= c- [c/b\b < a,0 < Ci := [c/b\b - [i[c/b\b/a)\a < 
2a — b and [c/b\ > 2. Assume that iSa,6,c 7^ cmd either a/b ^ Q or 
a/b = p/q and c/b G Z/g for some coprime integers p and q. Then 
T^a,b,c = ^/ o-nd only if 

(5.18) {[c/b\ + l)\Sa,b,c n [0, Co + a - &)| + [c/b\ \Sa,b,c n [Co, a) I = a. 

For the triple {a,b.,c) = (7r/4, 1,23 — ll7r/2) in Example 5.1, the 
equality (5.18) holds, and hence ^(x[o,23-ii7r/2)5 7rZ/4 x Z) is a Gabor 



frame even though the maximal invariant set Sa,b,c is not an empty 
set. For the triple (a,b,c) = (-\/3/2, 1, 15v^/2) with irrational ratio 
between a and b, ^(xfq ^^5^/2), a/3Z/2 x Z) is not a Gabor frame as 
(5.18) does not hold. In fact, in this case 

([12 - tVs, 7 - aVs) U [8 - 9^3/2, 3 - 3^3/2) 

U[4 - 2^3, 11 - 6^3)) + y3Z/2. 



Va. 



b,c 



Sa. 



b,c 
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For the triple {a,b,c) = (6/7,1,23/7) in Example 5.3 and the triples 
b c) = (13, 17, 77)/17 and (13, 17, 73)/17 in Example [slj the equal- 



[a 



ity (5.18) holds, but for the triple (13, 17, 75)/17 in Example 5.1 the 



equality (5.18) does not hold. Thus Gabor systems ^(x[o,23/7)5 6Z/7 x 
^); ^(X[o,77/i7), 13Z/17 X Z) and ^(x[o,73/i7), 13Z/17 x Z) are Gabor 
frames even though the maximal invariant sets are nontrivial, but the 
Gabor system ^(x[o, 75/17); 13Z/17 x Z) is not a Gabor frame. 

In this section, we finally consider the restriction of transformations 
Rabc and Rabc on the maximal invariant set Sabc- The finite-interval 



property for the maximal invariant set Sa,b,c in Theorems 5.4 and 5.5 
can also be interpreted as its complement consists of finitely many holes 
on a period. So we may shrink those holes into points, which maps 
the maximal invariant set Sa,b,c into the real line with marks. More 
importantly, after performing the above holes-removal surgery, the set 
J^a,b,c of marks on the line is a finite cyclic group if a/b = p/q and 
c G bZ/q for some coprime integers p and q, and the set /Ca,6,c of marks 
on the line is a finite subset of infinite cyclic group if a/b ^ Q, and 
the nonlinear application of the transformation Ra.b,c on the maximal 
invariant set Sa,b,c becomes a rotation on the circle M./Ya^b,c{ci-)'^ with 
marks ]Ca,b,c, see Appendix [C] for non-ergodicity of the piecewise linear 
transformation Ra,b,c- 

Theorem 5.8. Let (a, b, c) be a triple of positive numbers satisfying 
a < b < a, [c/b\ > 2,b — a < Cq := c — [c/b\b < a and < Ci :- 



c — Co — [(c — co)/a\a < 2a — b, and let Sa,b,c and Ya^^c be as in (2.9) 



and (2.16) respectively. Then the following statements hold. 

(i) // Sa,b,c 7^ o^nd either a/b ^ Q or a/b = p/q and c G bZ/q 
for some coprime integers p and q, then under the isomorphism 
Ya,b,c from Sa^b,c to the line with marks, the action to apply the 
piecewise linear transformation Ra,b,c on Sa,b,c becomes a shift 
on the line with marks; i.e., 

(5.19) 

Ya,b,c{Ra,b,c{t)+al.) = yaAc(^)+Xi,fe,c(ci-h&-a)+yaAc(a)^ for all t G Sa,b,< 

(ii) If Sa,b,c 7^ 0j ci/b = p/q and c G bZ/q for some coprime integers 
p and q, then marks on the line (i.e., images of the gaps in the 
complement of the set Sa,b,c under the isomorphism Ya^b,c) form 
a finite cyclic group generated by Ya^tj,c(yCi + b — a) + Ya.b,c[o)'Ij, 

}Ca,b,c = Ya,h,c{'^i + b-a)Z + Ya^bA(^)Z = gcd {Yafi^Ci+b- a) ,Ya,b,c{(^))'^^ 

where ICa^^c is the set of marks on the line. 



50 



XIN-RONG DAI AND QIYU SUN 



(iii) // Sa,b,c 7^ and a/b ^ Q, the set ]Ca,b,c of marks on the line is 
given by 

}Ca,b,c = U^UinYa^bAci + b-a) + Yaf>A<^)'L) 

•where M is the unique positive integer such that MYa^b,c{^i + 
6 - a) - Ya,b,c{co) e yaAc(a). 



In the next four subsections, we prove Theorems 5.4, 5.5, 5.6 and 



5.7, and 5.8 respectively. 



5.1. Maximal invariant sets with irrational time-frequency lat- 
tices. To prove Theorem 5^, we need a characterization of non-empty- 
set property for the maximal invariant set Sa^^c- 

Lemma 5.9. Let {a,b,c) be a triple of positive numbers satisfying a < 
b < c, b—a < Co := c— [c/b\b < a, [c/b\ > 2 and a/b Q. Then Sa,b,c 7^ 
if and only if there exists a nonnegative integer D < [a/{b — a)\ — 1 
such that (-Ra,6,c)'^([c — Co, c + 6 — Co — a) +aZ) = [co + a — 6, Co) +aZ and 
that (-Ra,6,c)"'([c— Co, c+6— cq— a)+aZ), < n < D — 1, has their closures 
being mutually disjoint and contained in (0, Co + a — 6) U (cq, a) + aZ. 

The sufficiency of the above lemma follows from the invariance of 
the set U^^o(-^a,b,c)"'([c — Co, c — Co + fe — a) + aZ) under transformations 
Ra,b,c and Ra,b,c, and the minimality of the set M\Sa,b,c in Theorem 



4.1 To prove the necessity in Lemma 5^, we let D be the minimal 
nonnegative integer such that (-Ra,6,c)'^([c — Cq, c + 6 — Cq — a) + aZ) fl 
([cq -|- a — 6, Co) + aZ) ^ 0. The existence of such an integer D follows 
from the mutually disjointness of holes {RafiA"'{[c — Co,c + b — co — a) + 
aZ),0 < n < D — 1 (and their closures), see (5.29). We then prove 



that {Ra,b,c) ii'^ — Co, c + 6 — Co — a) + aZ) = [co + a — 6, co) + aZ by 
applying Lemma 3.11 and Propositions 3.8 and 3.10, see (5.36). Now 



assuming that Lemma 5.9 holds, we start our proof of Theorem 5.4 



Proof of Theorem\5.4\ Assume that iSa,6,c 7^ 0- By Lemma 5.9, there 



exists a nonnegative integer D < [a/{b — a)\ —1 such that {Ra,b,c)^i{^~ 
Co,c + b — Cq — a) + aZ) = [cq + a — b, Cq) + aZ, and (i?a,fe,c)"([c — 
Co, c + 6 — Co — a) + aZ), < n < D — 1, have their closures being 
mutually disjoint and contained in (0, Cq + a — b) U (co, a) + aZ. Then 
U^^Q(i?a,b,c)"([c— Co, c— Co+&— a)+aZ) is minimal set that contains black 
holes of transformations Ra,b,c and Ra,b,c, and that is invariant under 
those transformations Ra,b,c and Ra,b,c- Here the invariance holds as 
(5.20) 
(^aAc)"([c-Co,c-Co + 6-a) + aZ) = (i?^^, 



\D-n 



([Co- 



5, Co J 



■aZ) 
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for all < n < D by Proposition |3. 7 and the property that 

(5.21) {Ra^b,cT{[c-co,c+b-Co-a] + aZ) C (0, Co + a-6)U(co,a) + aZ 
for all < n < D — 1. This together with Theorem 4A_ proves that 

(5.22) Sa,b,c = M\ U^=o (RaAcTilc ' Cq, c - Cq + b - a) + aZ) . 



Hence the conclusions (5.2), (5.3) and (5.4) follow from (2.13), (2.14) 
and (|K20l)-(|5:22l). 



D 



We finish this subsection with the proof of Lemma 5.9 

Proof of Lemma [Xg[ (<S=) By (3^), |(-RaAc)"([c - cq, c - cq + b - 
a) + aZ) n [0, a) < b — a for all < n < D, which together with the 
periodicity ( |2.13 ) of the transformation Ra,b,c implies that 

IAdH [0,a)| < {D + l){b-a) < a, 

where Ad = U^^o(-Ra,b,c)"([c — cq, c — cq + b — a) + aZ). Hence 

(5.23) Ad ^ M. 



By (|5.23|), the sufficiency reduces to proving that 

<Sa,b,c by Theorem 



(5.24) 

(In particular. 



D 



4.1 



and Proposition 4.3). 



Recall that the black hole [c— Cq, c—CQ+b—a)+aZ of the transformation 
Ra,b,c is contained in Ad, and that the black hole [cq + a — b, Cq) + aZ 
of the transformation Ra,b,c is also contained in Ad as it is same as 
(-Ra,b,c)^([c — Cq, c — Cq + b — a) + flZ) by the assumption. Then we 
obtain from (3.48) and (3.50) that 

(5.25) Ra,b,M\AD) C R\Ad. 



Similarly we get from (2.13) and (3.50) that Ra,b,c{t) ^ [c — cq, c — cq + 
b — a) +aZ and that Ra,b,c(t) ^ U^=i(-Ra,fe,c)"([c — Cq, c — Co + b — a) +aZ) 
for any t Ad- This implies that 

(5.26) Ra,b,M\AD) C R\Ad. 

Take any t e R\A d, th en iR a,b,cTit) and {Ra,b,cTit),n > 0, belong 
to M\y4£) by (5.25) and (5.26) and hence they do not fall in the black 
holes of the transformations Ra,b,c and Ra,b,c- Therefore t e Sa,b,c by 
Proposition 4.3, and hence (5.24) is established. 
(^^) First we need the following claim: 

Claim 5.10. There exists a nonnegative integer n < \_a/{b — a)\ — 1 
such that 

(5.27) {{Ra,b,cT{[c-co,c-co + b-a) + aZ))n{[co + a-b,co)+aZ) ^0. 
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Proof. Suppose, on the contrary, that 

(5.28) {iRa,b,cT{[c-Co,c-Co + b-a) + aZ))ni[co + a-b,Co) + aZ) = 

for all < n < [_a/{b — a)\ — 1. We first need to prove that 

(5.29) 

{RaAc)"{[c-co,c-co+b-a)+aZ))r]{Ra,b,cfi[c-co,c-co+b-a)+aZ)) =0 

for al l < n ^ n' < [a/{b — a)\ — 1. Suppose on the contrary that 

(5.29) does not hold. Then there exists t G ((-Ra,fe,c)"([c — cq, c — cq + 



b-a +aZ)) n {{Ra,b,cT'{[c-Co,c-Co + b-a) + aZ)). By ( |3.50| ) and 

), {Ra. 



( |5.28[ ), (i?aAc)"^"'^"'"'^"'W e (i?aAc([c-Co,c-Co + 6-a) + aZ))n([c- 
Co, c — Co + & — a) + aZ) , which is a contradiction as (-Ra,fe,c([c — Cq, c — 
Co + & — a) + aZ)) n ([c — Co, c — Co + 6 — a) + aZ) C ([c — Cq + &, c — 
Co + 2a ) + gZ ) n ([c - Cq, c - Cq + & - a) + aZ) = by ( |2.13[ ) and ( |5.28[ ). 
Hence (|5.29 ) is estabhshed. 

By (3.65) and (5.28), we have that 

(5.30) I {{Ra,b,cTi[c - Co, c - Co + 6 - a) + aZ)) n[0,a)\=b-a 
for all < n < [a/(6 — a)\ — 1. This together with (5.29) implies that 

a > 

+ \[co + a - b, Co)\ 

la/(b-a)\-l 

J2 I iiRa,b,cTi[c - Co, c - Co + 6 - a) + aZ)) n [0, a) 

n=0 

+ ib-a) 

([a/(6 - a)\ + l){b - a) > a, 



[a/(b-a)i~l 
■'n=0 



ur-n" '^" ^ {{Ra,b,c)"i[c - Co, c - Co + 6 - a) + aZ)) n [0, a) I 



> 



which is a contradiction. This completes the proof of Claim 5.10 D 



We return to work on the proof of the sufficiency. Let D be the 
smallest nonnegative integer such that 

(5.31) {{Ra,b,cf{[c-co,c-co + b-a) + aZ))n{[co + a-b,co) + aZ)^(lS. 

Then D < \_a/{b — a)\ — 1 by (5.27). By the definition of the integer 
D, we have that 

(5.32) ((i?„,6,e)"([c-co,c-co + 6-a) + aZ))n([co + a-6,co) + aZ) = 

for all < n < D — 1. Following the argument used to establish (5.29), 

we have that 

(5.33) 

{Ra,b,cT{[c-co,c-co+b-a)+aZ))^n(^{Ra,b,cT'{[c-co,c~co+b-a)+aZ))^ =0 
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ioT all <n ^ n' < D - 1. We claim that for all < n < D - 1, 

(5.34) {Ra,b,cT{[<^ - Co, c - Co + fe - a) + aZ) = [u„, m„) + aZ 

with 



(5.35) 



Ur 



Un + h — a and [m„, u„] C (0, co + a — h) or (cq, a); 



that is, (-Ra,fe,c)"([c — Co, c — Co + 6 — a) + aZ are periodic holes of length 
6 — a 'strictly' contained in (0, cq + a — 6) U (cq, a) + aZ. Let mq be the 
unique number in (0, a] such that c — co + 6 — a — mq G aZ, and -Uo < uq 
be the largest number such that c — co — mo G aZ. Then [c — co, c — co + 
6 — a) + aZ = [uq, uo) + aZ. Furthermore Mo > and mo < a as otherwise 
either ((0, e)+aZ)nSa^b,c = or ((— e, 0)+aZ)nSafi,c = for some small 
e > by Proposition |3. 10 which contradicts to the conclusion (3.55) 



in Lemma 3.11 This together with (5.32) and the definitions of Mq and 
Uq implies that Uq = Uq + b — a and [uq, Uq] C (0, Cq + a — b] or [co, a). 
Therefore it suffices to prove uq j^ cq + a — b and uo 7^ co. Suppose on 
the contrary that uq = Cq + a — b, then (_Ra^fe^c([0,e) + aZ)) fl Sa,b,c = 
{[uo,uo+e)+aZ)r\Sa,b,c C {[co+a—b, co)+aZ)nSa,b,c = for sufficiently 
small e G (0, co + a - b) b y KE^. Thus ([0, e) +'aZ ) n Sa,b,c = by 



(3.29) and Proposition 3.8, which contradicts to (3.55) in Lemma 3.11 
This proves that Uq ^ Cq + a — b. Similarly we can prove that Uq ^ Cq. 
Therefore the conclusions (5.34) and (5.35) hold for n = 0. Inductively, 
we assume that {Ra^b,c)^{[c — Co,c — Co + b — a) + aZ) = [{t„,M„) + aZ with 
Un = Un + b — a and [«„, u„] C (0, Co + a — 6) or (co, a). Then we see that 

{Ra,b,cT~^^ {[C - Co, C - Co + 6 - a) + aZ) = [Ra,b,cUn, Ra,b,cUn) + oZ with 

Ra,b,cUn — Ra,b,cUn = b — a. Let Un+1 be the unique number in (0, a] such 
that Ra,b,cUn — Un+1 G ttZ, and «„+! < M^+i be the largest number such 
that Ra,b,cUn — Un+i G flZ. Then (_Ra^fc^c)"'''^([c— co, c— co + 6 — a)+aZ) = 
[un+i,Un+i) + aZ. Similarly we can show that < Un+i < w„+i < a 
and ['U„+i,-u„+i)n [co + a — &, co) = 0, and hence m„+i = Un+i + b — a and 
[Un+i,Un+i] C (0,Co + a — 6] or [un+i,Un+i] C [co,a). Now we prove that 
Un+i ^ Cq + a — b and Un+i 7^ Co. Suppose on the contrary that Un+i = 
Cq + a — b. From the inductive hypothesis, we have that for < A; < n, 
(-Ra,6,c)*^^"^([0, 6)+aZ) = [Mfc,Mfc+e)+aZ where e > is sufficiently small. 



This together with (3.50), the inductive hypothesis and Proposition 3.8 
implies that ([0,e) + aZ) r\ Sa,b,c = 0, which contradicts to (3.55) in 



Lemma 3.11 This proves that Un+i 7^ cq + a — 6. Similarly we can 



prove that u„+i 7^ cq. Hence we can proceed the inductive proof of the 



conclusion (5.32) 



Next we prove that 

(5.36) (i?a,6,e)^([c-C 



Cq + b — a) + aZ) = [co + a — 6, Co) + aZ. 
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From ( |2.13[ ), ( |5.34[ ) and ( |5.35[ ) it follows that 

(5.37) (i?,,b,c)''([c-co,c-co + 6 

for some -U/j G (0, a] and Ud — Ud = & — a. By (5.31) 
6, Co) 7^ 0, which implies that ud > Co + a 



aZ) = [ud, Ud) + aZ 



UD,UD>^[Co + a- 

b. Suppose that Co + a — b < 



Ud < Co, then (i?a^b^c)^^^([0, e) + aZ) = [-u^i , tt/) + e) + aZ for sufficiently 



small e by (5.34) and (5.35). This together with (3.50), (5.34) and 



(5.35) and Proposition 3.8 implies that ([0, e) + aZ) fl Sa,b,c = 0, which 
contradicts to (3.55) in Lemma [3.11 Therefore ud > cq. Similarly, we 
can prove that ud < co + a — b. Hence (5.36) follows as ud — ud = b — a. 
Finally we prove that {Ra,b^c)"'{[c—co,c—co + b — a)+aZ),0 <n<D, 
have their closures being mutually disjoint 
have that 



By Proposition 3.10 



we 



(-Ra,fe,c)"([c - Co, c - Co + 6 - a) + aZ) n Sa,b,< 



0<n< D. 



From (5.34), (5.35) and Lemma 3.11, we obtain that 

{un - e, Un) n Sa,b,c 7^ and {un, u„ + e) n Sa,b,c 7^ 0, <n < D 
for any sufficiently small e > 0. Combining the above two observations 



with (5.33)-(5.36) prove mutual disjointness for the closures of holes 
iRa,b,cT{[c - Co,c - Co + b - a) + aZ),0 < n < D. D 

5.2. Maximal invariant sets with rational time-frequency lat- 



tices. To prove Theorem |5.5| we need the following result about the 
maximality of the set Sa,b,c under the transformation Ra,b,c in the case 
that a/b G Q, c.f. Theorem 4.1 for the maximality of the set Sa,b,c 
under two transformations Ra.b,c and Ra,b,c in the case that a/b ^ Q. 

Lemma 5.11. Let a < b < c,b — a < Cq := c — [c/b\b < a, [c/b\ > 2, 

a/b = p/q and c G bZ/q for some coprime integers p and q. Then 
(i) Sa^b,c is the maximal set that is invariant under the transforma- 
tion Ra.b,c O'lT'd that has empty intersection with the black hole 
[co + a — &, Co) + aZ of the transformation Ra,b,c- 
(ii) ]R\iSa,6,c is the minimal set such that it is invariant under the 
piecewise linear transformation Ra,b,c and that it contains the 
black holes of piecewise linear transformations Ra,b,c o,nd Ra,b,c- 



To prove Theorem 5.5, we also need the following result about the 
dense property of the maximal invariant set Sa^^c around the origin in 
the case that a/b G Q, c.f. Lemma 3.11 for the dense property of the 
set Sa,b,c in the case that a/b ^ Q. 

Lemma 5.12. Let a < b < c,b — a < cq < a, \_c/b\ > 2, a/b = p/q and 
c/b G Z/q for some coprime integers p and q. Assume that Sa,b,c 7^ 0- 
Then the following statements hold. 



THE abc-PROBLEM FOR GABOR SYSTEMS 



55 



111 



At least one of two intervals [0, b/q) and co + [0, b/q) is contained 

At least one of two intervals [—b/q^ 0) and cq + a — 6 + [—b/q^ 0) 

is contained in Sa,b,c- 

At least one of two intervals [0,b/q) and [—b/q,0) is contained 

'ITT' <Sa^b,c- 



Now supposing that Lemmas 5.11 and 5.12 hold, we start the proof 
of Theorem 5^ c.f. Examples |5.2| and |5.3| in Section [5j 



Proof of Theorem |5.4 Let 5 G [0,co + a — b] and 6' G [cq — a, 0] be so 
chosen that [6', 6) is the maximal interval contained in M\Safi,c- Then 
6, 6' G bZ/q by (5.6), and they satisfy (5.8) by the assumption Safi,c 7^ 
and Lemma 5.12, since at least one of two intervals [0, b/q) and [—b/q, 0) 
is contained in Sa^^c- Therefore the first conclusion (i) holds. 

Now we divide the following three cases: (1) 6' = and 6^0; (2) 
6 = and 6' ^ 0; and (3) 6 = 6' = 0, to verify the conclusions (ii)-(vi). 

Case 1 5' = and 5^0 
In this case, 



(5.38) 
Then 

(5.39) [c 



[-b/q, 0) C Sa,b,c and [0, 6) H Sa., 



b,c 



Co - b/q, C-Cq) = Ra,b,c[-b/Q, 0) C Ra,b,cSa,b,c = Sa,b,c 



by (2.13), (5.38), and Proposition 3.8; 



(5.40) 



C 



[c — co + b — a + 6,c — co + b — a + S + b/q) 

Ra,b,c[5^ 5 + b/q) — a ifO<(5<Co + a — 6 
-Ra,6,c[co, Co + b/q) li 5 = Co + a-b 

^a,b,c^a,b,c ^a,b,c 



by (2.13), (5.38), Proposition 3.8, the maximality of the interval [0,(5) 
in M\Sa,b,c) and the first conclusion in Lemma 5.12 and 



(5.41) 



[C - Co, C - Co + & - a + (5) n Sa,b,c 

(-Ra,6,c[-a, 5 - a) U [C - Co, C - Co + 6 - a)) n Sa,b,c 
^a,6,c([-a, 5-a)n Sa,b,c) = 0, 



where the first equality follows from (2.13), Proposition 3.8 and the 
assumption [0,6) C [0,co + a — 6), and the second equality holds by 
Propositions |3^ and [3]7 Thus [c — Cq, c — Cq + b — a + S) is a gap (i.e., 
an interval with empty set intersection with Sa^b,c) with length b — a + 6 
and boundary intervals of length b/q at each side in the set Sa^h^c- 
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Proof of the conclusion (ii). Let Ni be the smallest nonnegative 
integer such that (-Ra,6,c)^^ ([c — cq, c + h — cq — a + 5) + oL) fl ([cq + a — 
6, Co) + oL) 7^ if it exists, and Ni = +oo otherwise. Mimicking the 
argument used in the proof of Claim 5.10[ we have that A^^i < oo. We 
divide two cases to prove the conclusion (ii). 

Case la: A^i = 0. 



In this case it follows from (5.39), (5.40) (5.41) that {Ra,b,c)^'[c 



Co, c — Co + b — a + 5) = [c — co, c — co + b — a + 6) is a gap of length 
b — a + 6 with boundary intervals of length b/q at each side in the set 
Sa^b,c- This, together with [co,Co + b/q) C Sa^b,c by Lemma 5.12, and 
the definition of the nonnegative integer A^^i, proves that 



(5.42) (R, 



a,b,c) 



\Ni\ 



-Co,c — Co + b — a + 6) + aZ= [cq- 



-b — S,Co) +aZ 

and hence the conclusions (5.9) and (5.10) in the case that A^^i = 0. 

Case lb: 1 < iVi < oo. 
In this case 

(5.43) (i?aAc)"([c-co,c+6-co-a + (5) + aZ)n([co + a-6,co)+aZ) = 
for all < ra < A^^i — 1. Let us verify that 

(5.44) {Ra,b,c)''i[c-co,c+b-co-a + S)+aZ) = [bn + a-b-5,bn)+aZ 
with 

(5.45) [bn + a-b-6 - b/q, 6„ + b/q) C [0, Cq + a - 6) U [cq, a). 



(5.46) 
and 



{[br, + a - b - 5,bn) + aZ) n Sa 



:b,c 



b,c 



(5.47) [bn + a-b-6-b/q,bn + a-b-6) + aZ,[bn,bn + b/q) + aZ C Sa. 

for all < n < A?"! — 1. For n = 0, write (-Ra,6,c)"([c — cq, c + b — co — a + 
6) + aZ) = [c — Cq, c + b — Co — a + 6) + aZ = [bo + a — b — 6, bo) + aZ with 
bo G (0,a]. Then t he co n clusio ns (|5.44D , (|5.45D , ( |5.46|) a nd ( |5.47[ ) for 
n = follow from ( [33o| ), ( [K39| ), |5^0 r (IS^ITa nd (|5.43D . Inductively 
we assume that the conclusions (5.44), (5.45), (5.46) and (5.47) hold 



for all < n < fc < A'^i — 2. Then ioi n = k + 1, 

(-Ra,6,c)"([c - Co, c + 6 - Co - a) + aZ) 



Ra,bA^k + a-b- 6,bk) + aZ (by ( |5.44[ ) with n = k) 
[Ra,b,c{bk + a-b- 5), Ra,h,c{bk + a-b-5) + b-a + 5) + aZ 



(by (5.45) with n = k) 
[bk+i + a-b-6, bk+i) + aZ 
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([6fc+i + a-b-6, bk+i) + aZ) n Sa,b,c 
= Ra,b,c{i[bk + a-b- 6,bk) + aZ) n Sa,b,c) 
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(by (5.43), (5.45) for n = k, and Proposition 3.7) 



[bk+i + a-b- 5 - b/q, bk+i + a - b - 5) + aZ 
= [Ra,b,c{^k + a-b-6) - b/q, Ra^A^k + a-b-6)) + aZ 
= Ra,b,c\bk + a-b- 6 - b/q, bk + a - b - 6) + aZ 

(by (5.46) with n = k) 
= (i?a,6,c)'+' [c - Co - b/q, c - Co) + aZ C Sa,b,c (by ( p9| )) 
and similarly 

[bk+i,bk+i + b/q) + aZ 



{Ra,. 



b,c 



\k+l 



[c — CQ + b — a + 6,c — Co + b — a + 6 + b/q) + aZ C. Sa,b,c 



by ( |5.40[ ), ( |5.44[ ), ( |5.45| ) and ( |5.47[ ) for n = fc, the definition ( |2.13[ ) 
of the transformation Ra,b,c, and the invariance (3.30) of the set 5a^fe^c 
under the transformation Ra,b,c- This together with (5.43) completes 
the inductive proof of ( |5.44[ ), ( |5.45D , ( |5.46[ ), and ( |5.47[ ). 

Recall that (-Ra,6,c)^4c — cq, c — co + 6 — a + (5)n[co + a — 6, co)+aZ 7^ 



and [cq, Co + b/q) + aZ C Sa,b,c by the first conclusion in Lemma 5.12 



Then applying (5.44) and (5.45) with n = Ni — 1, we obtain that 

(5.48) {Ra,b,c)^^['^~'^OTC — Co + b — a + 5)+aZ= [co + a — 6 — 5, Co)+aZ, 
because 

(5.49) {Ra,b,c)'''{[c -co,c-co + b-a + 6) + aZ)n Sa,b,c 

= Ra,b,c{i[^Ni-i + a-b- 5, 6Afi_i + a- a) + aZ) n Sa,b,c) = 
and 

(5.50) [cq + a — b — S — b/q, cq + a — b — S) + aZ 

= RaMil^Ni^i -b + a- b/q, bN^^i - b + a) + aZ) C Sa,b,c- 
Notice that 

(5.51) [&„, bn + b/q) + aZ = -Ra,b,c([&n-i, &n-i + b/q) + aZ 
= ■■■ = {Ra,b,c)'"{c -Co + b-a + 6) + [0, b/q) + aZ. 



The conclusion (ii), particularly the equalities in (5.9) and (5.10), in 
the case that 1 < A^i < cx) then follows from (|5.44l), (|5.48l) and (|5.51|). 
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Proof of the conclusion (in). Let N2 be the minimal nonnegative 
integer such that {Ra,b,c)^^{[co+a-b-6,co+a-b)+aZ)n{[0,6)+aZ) ^ 
if it exists and 1^2 = +00 otherwise. Mimicking the argument used to 
prove Claim 5.10 we have that N2 < oo. We divide three cases to 
verify ( [5ll| and ( [5l2| . 

Case Ic: N2 = 0. 



In this case, it follows from (5.38) and (5.50) that 

(5.52) [co + a-b-6,co + a-b) + aZ=[0,6) + aZ, 

and hence (5.11) and (5.12) hold for N2 = 0. 

Case Id: N2 = 1. 

In this case, we have that 

(5.53) -Ra,fe,c[co + a — b — 6, Co + a — b) + aZ = [bi — 6,bi) + aZ 

for some 61 G (0, a] with bi — 6 — Ra,b,c{co + a — b — 6) G aZ. Recall that 
[co + a- b -^- b/q, Cq + a - b-S) C Sa^b^c and [cq, Cq + b/ q) C 5a,fe,c 
by (5.38), (5.50) and Lemma 5.12, we then obtain from (5.53) and 
Proposition 3.8| that 

(5.54) [bi — 6 — b/q, bi — 6) + aZ C Sa^^c and [61, bi + b/q) +aZ C Sa,b,c- 

Therefore [bi — 6,bi) is a gap of length S with boundary intervals of 
length b/q at each side in the set Safi,c- Thus 

[61 - 6, 61) n ([co + a - 6, Co) + aZ) = 

as the gap containing [co+a—b, cq) is (-Ra,6,c)^^ [c— Cq, c— Co+&— a+(5)+aZ 
which has length b — a + 6 and boundary intervals of length b/q at 
each side in Sa^^c- By the definition of the nonnegative integer A^25 
([61 - 6, 61) + aZ) n ([0, 6) + aZ) ^ 0. This together with ( |5.54| ) and 
[— 6/g, 0) G iSa^b^c implies that bi = 6 and 

(5.55) -Ra,6,c[co + a — 6 — (5, Co + a — 6) + aZ = [0, 5) + aZ. 



The conclusion (5.11) and (5.12) for A^2 = 1 follow from (5.53) and 
( |K55| . 

Case le: 2 < iV2 < +cx). 

In this case, following the arguments for the first conclusion of this 
theorem and the arguments in the case that A^2 = 1, "we may inductively 
show that 
(5.56) 
iRa,b,cTi[co+a-b-6,Co+a-b)+aZ) = [6„-(5, 6„)+aZ, 1 < m < N2-I, 



for some b^ G (0,a] with [bm~S — b/q,b. 
[bm- 5 - b/q, K 



-b/q) C [0,Co + a-6)U[co,a) 



^) C Sa,b,c and [bra.bm + &/g) C Sa,b,c- Applying 
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(5.56) with m = N2 — 1, and recalling the definition of the integer N2 



we obtain that 



(5.57) (i?a,fe,c) '([co + a - b - 6,Co + a - b) + aZ) = [0,6) + aZ 
and 

(5.58) [-b/q,0),[5,5 + b/q)eSa,b,c. 

Therefore the conclusions (5.11) and (|5.12[) for 2 < A^2 < 00 follow 



from (5.56), (5.57) and (5.58). 



Proof of the conclusion (iv). First we prove the mutually disjoint 
property for (-Ra,fe,c)"([c — cq, c + b — cq — a + 6) + aZ), < n < Ni, i.e., 
(5.59) 
iRa,b,cT'i[c-co,c+b-co-a+6)+aZ)n{Ra,b,cT'i[c-co,c+b~co-a+6)+aZ) 

for all < ni y^ n2 < Ni. By ( |5.44D -( [K50| and Proposition |3.10 
{Ra,b,c)"'[c—co,c—co+b—a+6),0 <n< Ni, are gaps with length 6— a+(5 
and boundary intervals of length b/q at each side contained in the set 
Safi,c- Then for any < ni 7^ 712 < Ni, either {Ra,b,c)"'^ {[c — co,c + b — 
Co-a+6)+aZ) = {Ra,b,cT'{[c-Co,c+b-Co-a+5)+aZ) or {Ra,b,cT' {[c- 
Co,c + 6-Co-a + 5) + aZ)n(i?aAc)"'([c-Co,c+6-Co-a + (5) + aZ) = 0. 
Suppose that {Rafi^c)"'^{[c — Co,c + b — CQ — a + 6) +aZ) = {Ra,b,c)"'^ {[c — 
Cq, c+b — CQ — a + 6) + aZ) for some < ni < n2 < Ni. Then n2 < Ni — 1 



by (5.43) and (5.48). Thus by (5.43) and the one-to-one correspondence 
of the transformation Ra,b,c on the complement of [cq + a — b, Cq) + aZ, 



Co,C- 



Cq + b — a + 6) + aZ = [c 



a + S) + oL, which contradicts to the range property (3.49) as 



we have that (i?a,6,c)"'"'''[c 

co + & 

((^a,6,c)"'""' [c - Co, c - Co + 6 - a + 5) + aZ) n ([co + a - 6, co) + aZ) = 
by (5.43). This proves (5.59). 

Next we verify that (i?a,fe,c)™([co + a — & — (5, co + a — 6) + aZ), < m < 
N2, are mutually disjoint when 1 < A^2 < 00. Recall that (-Ra,fe,c)™([co + 
a — 6 — (5, Co + a — 6) + aZ) , 1 < "m < A^2 , are gaps of length b with 
boundary intervals of length bjq on each side contained in the set Saf,^c 
by (5.55), (5.56) and (5.57). Therefore any two of gaps (-Ra,b,c)™'([co + 



a — 6 — (5, Co -|- a — 6) + aZ), \ < m < N2, are either identical or has 
empty-set intersection. If there exist 1 < nii < 1712 < N2 such that 
gaps (-Ra,6,c)™'([co + a ~ b - 5, Cq + a - b) + aZ) and (-Ra,6,c)™'([co + a- 
b — 6,cq + a — b) + aZ) are identical, then the front parts of those gaps 
of length min(5, b — a) should be identical too, i.e., (-Ra,fe,c)™^([co + a — 
b — 5,Cq + a — b — 5 + min(5, 6 — a)) + aZ = (-Ra,fe,c)'"^([co + a — b — 
5,CQ + a — b — 5 + min(5, b — a)) + oL. Recall from (5.42) and (5.48) 
that (-Ro,6,c)™'([co + a — b — 5,CQ + a — b — 5 + min(5, b — a) + aZ) = 
(^a,fe,c)'"+^'([c - Co, c - Co + mm{6,b- a)) + aZ),0 < m < N2 and 
(-Ra,6,c)"([c-Co,c-Co + min((5, &-a)) + aZ) fl ([co + a-6, Co) + aZ) = 
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for all < n < A^^i + A^2- Therefore by the one-to-one correspondence 
of the transformation Ra,b,c on the complement of its black hole in 

Co + min((5, b — a)) + aZ = 



Proposition 3.7 



(i?a,M)"'""1c 



Co,C 



[c— Co, c— co-|-min(5, 6— a))+aZ, which contradicts to the range property 
(3.49) of the transformation Ra,b,c- This proves that (-Ra,6,c)™([co + a — 
b — 6,Co + a — b) + aZ), 1 < m < A^2, are mutually disjoint. 

From the above argument, we see that {Ra,b,c)"'{[c — Co,c — Cq + b — 
a + 6) + aZ), < n < Ni, are mutually disjoint, and that {Ra,b,c)"^{[co + 
a — b — S, Cq + a — b) + aZ), 1 < m < N2, are mutually disjoint. Then 
verification of the mutually disjoint property in the conclusion (iv) 
reduces to showing that any gap of the form (-Ra,6,c)"([c — Co, c — Co + 
b — a + 6) + aZ), < n < Ni, has empty intersection with any gap of 
the form {Ra,b,c)"' {[cq + a - b - 6,co + a - b) + aZ),l < m < N2. This 
is true because {Ra,b,c)^[c — Co,c — Co + b — a + 6),0 < n < Ni — 1, are 
gaps of length b — a + 6 with boundary intervals of length b/q at each 
side contained in the set Sa^b,c by (5.44)-(5.50) and {Ra,b,c)"^{[co + a — 



b — 5,Co + a — b) + aZ) ,1 < m < N2, are gaps of length 6 with boundary 



intervals of length b/q on each side contained in the set Sa,b,c by (5.55) 
(IKsel and K57\). 



Proof of the conclusion (v). Writ e 6 = l{b—a)+S for some < / G Z 
and 6 E (0,6 — a]. From (5.9)-(5.12), we obtain that 



{Ra,b,cTi[<^ - co,c- co + b-a) + aZ) 

K-i(N^+N2+i) + ~Kb - a) - (5 + [a - 6, 0) + aZ 

if < n - l{Ni + N2 + l)<Ni for some < / < /, 
K-i{N,+N2+i)-N, + K^ - a) - 5 + [a - 6, 0) + aZ 

if 1 < n - /~(A^i + N2 + l)-Ni<N2 for some </"</- 1, 

{(bn~i{N^+N2+i)-Ni + [-^, 0)) U [co + a - 6, Co - 5)) + oL 
if 1 < n - i{Ni + N2 + l)-Ni< N2, 

{ibn-{i+i)(Ni+N2+i) + [-^, 0)) U [co + a - 6, Co - 6)) + aZ 
if < n - (/ + l){Ni + N2 + l)< Nu 



where 6„ = {Ra,b,c)"'{c — Cq + b — a + 6),0 < n < Ni, and 6^ 



{Ra,b,c)"^{co), I < m < N2, c.f. Example 5.2 in Section pi Therefore 
(U^io((^a,6,c)"[c - Co, c - Co + 6 - a + 5) + aZ) 



U(U^Li(i?a,6,c)"'[co + a-b-6,Co + a-b) + aZ) 
{[c — Cq, c — Co + b — a + 6) + aZ) 



.N1+N21 



I |iyi-rJV2 / p 

U„_n [-tla,b,c 



(5.60) 



u 



il+l){Ni+N2+l)+N-_ 
n=0 



' (-Ra,fe,c)"[c - Co, C - Cq + 6 - a) + ttZ, 
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and 

(5.61) 

{Ra,b,cf^^^'-^'^'''^^^^^'[c-co,c-co + b-a)+aZ=[co + a-b,co)+aZ. 

Hence the union of the gaps {{Ra,b,c)"'[c — co,c — co + b — a + 6)+aZ),0 < 
n < Ni, and (-Ra,b,c)'"[co + a — b — 6,co + a — b) + aZ), 1 < m < N2, is 
invariant under the transformation Ra,b,c and contains the black holes 
of the transformations Ra,b,c and Ra,b,c- It also indicates that any points 
not in that union will not be in that union under the transformation 



Ra,b,c- This together with Lemma 5.11 proves (5.13) and hence the 
conclusion (v). 

Proof of the conclusion (vi). From the arguments to prove the con- 
clusions (ii) and (iii), the boundary intervals of the mutually disjoint 
gaps ((-Ra,6,c)"'[c — Co, c — Co + & — a + 5) + aZ),0 < n < Ni and 
(-Ra,b,c)'^[co + a — 6 — (5, Co + a — 6)+ aZ), 1 < m < A^2, of length b/q at 
each side are contained in the set Sa,b,c- Therefore the set Sa,b,c is the 
union of intervals [6„, 6„ + /i„) + aZ, {) <n < Ni and \bmi bm + ^m) + ^Z, 
where < /i„ G 6Z/g, < n < iVi, and < /i^ G aZ/g, 1 < m < A^2, 
are chosen so that \bn + hn,bn + hn + b/q) + aZ, < n < Ni and 
[6m + hm,bm + hm + b/q) + aZ, 1 < m < A^2, are contained in gaps. As 
[0, 5) + oL and [cq + a — 6, cq) + aZ are contained in the union of the 
mutually disjoint gaps, each of the intervals [6„, 6„ + hn) + aZ, < 
n < Ni, and [bm,bm + /im) + aZ, 1 < m < A''2, is contained ei- 
ther in [0, Cq + a — 6) -|- aZ or in [cq, a) + aZ, and its boundary in- 
terval of length b/q at each side is not contained in the set Sa,b,c- 
Recall that bn - {Ra,b,c)''{c - cq + b - a + 6) G aZ, < n < Ni, 
and bm - (i?a,6,c)'"+^'(c - Co + b- a + 5) G aZ, 1 < m < A^2, from 
the second and third conclusions of this theorem._ Hence the interval 

[bn, bn + hn) + oL = (-Ra,6,c)"[6o, 60 + ^o) + oX and [6m, bra + /^m) + oL = 

{Ra,b,c)"^~^'^^ [bo, bQ+hQ)+aZ. This together with the measure-preserving 



property in Proposition 3.8 implies that the length of intervals con- 



tained in the set Sa,b,c are the same, i.e., 

(5.62) hn = hm = h for all < n < A^i and 1 < m < iV2 



where < /i G aX/q. Hence (5.14) holds. Finally we verify (5.15). 



Note that the measure of the gaps contained in [0, a) is equal to {Ni + 
l){b — a + S) + N26, while the measure of the intervals contained in 
Sa,b,c n [0, a) is (A^i + N2 + l)h. Therefore 

(5.63) {Ni + N2 + l)h +{Ni + l){b-a + 6) + N26 = a. 



Thus (5.15) follows. This completes the proof of the conclusion (ii)-(vi) 



for the case that S' = and 5^0. 
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Case 2 S = and S' ^ 0. 

We follow the argument used for Case 1 to prove the desired conclu- 
sion. We omit the details here. 

Case 3 5 = 6' = 0. 

In this case, we have that 

(5.64) [-b/q,b/q)cSa,b,c. 

Then 

[c-Co-b/q,c-Co) C Sa,b,ca^d[c+b-Co-a,c + b-Co-a + b/q) C Sa,b,c 

as [c — Co — b/q, c — cq) = Ra,b,c[~b/q, 0) and [c + b — co — a, c + b — 
Co — a + b/q) = Rafi,c^,b/q) and the set Sa,b,c is invariant under the 



transformation Ra,b,c by Proposition 3.8 



Proof of the conclusion (ii). Let A^i be the smallest nonnegative 
integer such that (-Ra,6,c)^^ ([c — cq, c + 6 — cq — a) + aZ) fl ([cq + a — 
b, Co) + aZ) 7^ if it exists, and Ni = +oo otherwise. Following the 
argument in Case 1, we can show that A^^i < +oo, and 
(5.65) 
(-Ra,6,c)"([c — Co, c + 6 — Co — a) +aZ), <n< A''i,are mutually disjoint, 

(5.66) (-Ra,b,c)"([c — Co, c + 6 — Co — a) + aZ) = [6„ — 6 + a, 6„) + aZ 

for some 6„ G (0, a) with 6„ — (-Ra,6,c)"(c + 6 — cq — a) G aZ, [6„ — 6 + 
a - b/q, bn + 6/g) C [0, Cq + a - 6) U [cq, a), 

(5.67) [6„ + a - 6 - b/q, 6„ + a - 6) + aZ, [6„, 6„ + b/q) + aZ C 5a,fe,c 
for < 72 < A^^i, and 

(5.68) \bNi_ + a-b,bNj = [cq + a - 6, cq). 

Therefore the conclusion (ii) follows. 

Proof of the conclusion (Hi/ . Let N2 be the smallest positive integer 
such that (-Ra,b,c)^^(co) e aZ if it exists and N2 = +cxd otherwise. To 
prove (iii)', it suffices to prove that 

(5.69) N2 < +00. 



To prove (5.69), we need the following mutually disjoint property when 

A^2 < 00: 

(5.70) 

{Ra,b,c)"'{co) + [-b/{2q),b/{2q))+aZ,0 < m < N2, are mutually disjoint. 
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Proof. Suppose, on the contrary, that the mutual disjoint property in 



(5.70) does not hold. Then there exist 1 < mi < m2 < N2 such that 

(5.71) {Ra,b,cr' (Co) + aZ= {RaMr' (Co) + aZ. 

This implies that m2 < N2 by the definition of the integer A^2- Recall 



that Co e Sa,b,c by applying (5.67) and (5.68) with n = Ni. Then 



{Ra,b,c)"^{co) € <Sa,b,c by Propositiou 3.8 This, together with the one 



to-one correspondence for the transformation Ra,b,c onto Sa^^c given in 



Proposition |3.8[ leads to 

(5.72) Co + aZ 



iRa,b,cr'-"''ico) 



oL 



by (5.71). Recall that co G {Ra. 



fe,c)^^^"'^(0) + aL. Then applying the 
one-to-one correspondence of the transformation Ra^b.c on the invariant 
set Sa}>,c again, we obtain from (|5.72[) that (i?a,6,c)™'"™'(0) e oL. This 



is a contradiction as m2 — mi < Ni — 1 and iVi -|- iV2 + 1 is the smallest 
positive integer n such that (-Ra,b,c)"(0) G aZ. D 

Now we prove that N2 < p — 1. Suppose on the contrary that 



A^2 > P- Following the argument in the proof of (5.70), we obtain 



that (i?a,b,c)"'(co) + [-b/{2q),b/{2q)) + aZ, < m < p are mutually 
disjoint. This is a contradiction as there are at most p elements in the 
set [0, a) n bZ/q, and hence proves the conclusion (iii)'. 



Proof of the conclusion (iv)' . By (5.65) and (5.70), verification of 



the mutually joint property in the conclusion (iv)' reduces to showing 
that (i?a,6,c)™(co) ^ U^io(^a,fe,c)"([c - Co, c - Co + 6 - o) + aZ) for all 
< m < A^2, which is true as (-Ra,6,c)™'(co) € Sa,b,c for all m > and 
{Ra,b,cT{\c - Co, c - Co + 6 - a) + aZ) n Sa,b,c = 0'for all < n < A^i. 



Proof of the conclusion (v). We can follow the argument in the first 
case 6 > and S' = 0, as U^Iq (i?a,6,c)"'([c — cq, c — cq + 6 — a) + aZ) is 
invariant under the transformation Ra,b,c and contains the black holes 
of the transformations Ra,b,c and Ra,b,c- We omit the details here. 



Proof of the conclusion (vi). By (5.66)-( [5.68[ ), gaps of length b — a 
are (-Ra,b,c)"(c + b — cq — a) + [a — b,Q) + aZ, Q < n < Ni, while gaps 
of length zero are located at (-Ra,fe,c)"'(co) + aZ = (-Ra,6,c)™^^^(c + b — 
Co — a) + aZ, 1 < m < N2- By the conclusions (iii)' and (v), we can 
divide the set Sa,b,c as the union of disjoint union of intervals who are 
left-closed and right-open such that each interval has its left endpoint 
being the same of the right endpoint of a gap, each interval has its right 
endpoint being the same as the left endpoint of a gap, and each interval 
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has its interior not containing the location of any gaps of length zero; 
that is 



(5.73) 
where 
(5.74) 
and 



Sa. 



b,c 



u. 



N1+N2 

n=0 



[6„, bn + K) + aZ, 



hn e {RaAcTic + b-Co-a) + aZ 



(5.75) brr + h^e {6„, +a-b + oLYll^^ U {(i?aAc)™Hco) + ol.}Z=^ 
for all < n < A^i + N2, and 



(5.76) (i?a,6,c)'"(co) ^ U^ir^ 



(6„, 6„ + /i„) + aZ 



for all 1 < 771 < A^2- Therefore the proof of (5.14) reduces to establish- 
ing 

(5.77) hn = ho, 0<n<Ni + N2. 



By the measure-preserving property in Proposition |3.8| it suffices to 

prove that 

(5.78) 

[bn+i,bn+i + hn+i) + aZ C Ra,b,c{['^n,bn + hn) + aZ),0 <n< N1 + N2-I 

and 

(5.79) 

[bn,bn + K) + aZ C Ra,bA[bn+i,bn+i + hn+i) + aZ),0 <n < N1 + N2-I. 



By (5.74) and the measure-preserving property in Proposition 3.8 we 
have that Ra,b,c{[^n, bn + hn) + aZ) = [6„+i, 6„+i + hn) + aZ. Observe 
that for each < ra < A^^i + A^2, &n+i + hn is the left endpoint of a gap 
because 



Jn+l 



+ hn 



Ra,b,c{bn + hn) if bn + hn ^ {0, Cq + a - 6} -^ aZ 

-Ra,6,c(co) if 6„ + /i„ G Co + a - 6 + aZ, 



and bn + hn ^ b^-^^^^ + hNi+N2 + aZ = aZ for all < n < iVi + iV2 — 1 
by the conclusion (iv)'. This together with (5.76) and the fact that 
Ra^bAl^n, bn + h n) + aZ) C Sa,b,c provcs ( |5.78[ ). 

Si mila rly, by (5.74) and the measure-preserving property in Proposi 
tion 



3.8 



we have that 0<?2<A^i+A^2-l, -Ra,6,c([&n+1, &n+l + ^n+l) + 

aZ) = [bn, bn + hn+i) + oL and 6„ + /i„+i is the left endpoint of a gap 
because 



bn + ^n+1 



-Ra,6,c(&n+1 + /in+l) if &n+l + /^n+l ^{c,C- Cq} + oZ 



^7V, 



if bn+\ + /in+1 G c + aZ, 
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and bn+1 + hn+i ^ c— co + aZ = bo — b+a+alj for all < n < A''i+A^2 — 1 
by the conclusion (iv)'. Thus (5.79) follows. This proves ( |5.77[ ) and 
hence (5.14). 

The equation (5.15) holds by (5.14) and the conclusion (iv) in this 
theorem. D 

Now it remains to prove Lemmas |5. 11 and 5.12 

Proof of Lemma 5.11\ (i) Clearly it suffices to prove that 

(5.80) Sa,b,c = (i?a,6,c)''M\([co + a - 6, cq) + aZ) 
for some nonnegative integer D. For n > 0, write 

(5.81) {Ra,b,cT^\i[co + a-b,Co) + aZ) = A„ + [0, b/q) + aZ 

where A^ C {0, b/q, . . . ,{p — l)6/g}\[co + a — b, Cq). The existence of 
such a finite set An follows from the assumption on the triple (a, b, c) 
and the definition (2.13) of the transformation Ra,b,c- Clearly, 

(5.82) An+i cAn,n> 0. 

Notice that there are at most 2p — q elements in the set {0, b/q, . . . ,{p — 
l)6/g}\[co + a — b,Co). This together with (5.82) implies that 

(5.83) Ad+1 = Ad 

for some nonnegative integer D < 2p — q. Without loss of general- 
ity, we assume that D is the smallest nonnegative integer such that 

(5.83) holds. By Propositions 3.8 and 3.6, we have that Sa^b,c = 

{Ra,h,c)^Sa,b,c C {Ra,b,c)°^ and Sa,b,c H ([cq + O - 6, Cq) + oZ) ' = 0. 

This leads to the following inclusion 

(5.84) Sa,b,c^AD + [Q,b/q) + aZ. 



Clearly the conclusion (5.80) follows in the case that A^ 
we consider the case that Ad ^ and prove 

(5.85) AD + aZd Sa,b,c- 



Now 



For n > 0, it follows from (2.13) and (5.81) that 
An+i + [0, b/q) + aZ 

= Ra,b,c{{Ra,b,cT^\i[CO + a - 6, Cq) + aZ))\([co + a - 6, Cq) + aZ) 

= Ra,b,Mn + [0, b/q) + aZ)\([co + a-b,Co) + aZ) 
= Ra,b,Mn + aZ)\([co + a-b,Co) + aZ) + [0, b/q). 

Therefore 

(5.86) An+i + aZ = Ra,b,Mn + aZ)\{[co + a - b,Co) + aZ), n> 0. 
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Applying (5.86) with n = D and using (5.83), we obtain that 

Ad + aZ = Ad+i + aZ = Ra,b,c{^D + aZ)\([co + a - 6, Cq) + aZ). 
Thus 

(5.87) Ra,b,MD) + aZ = Ad + aZ 

as the cardinahty of the sets A^ and Ra^^dAD + aZ) fl [0, a) are the 
same. This together with (3.49) and (5.86) imphes that 

(5.88) {Ad + aZ)n{[c-co,c + b-co + a) + aZ) = 0. 
Hence 

(5.89) Ra,b,MD) + aZ = Ad + aZ 



by (5.81), (5.87), (5.88) and Proposition KTS 



Take t G Ad- Then {RgM cTit) and {Ra,b,cTit),n > 0, belong to the 
set Ad + aZ by (5.87) and (5.89), and hence they do not belong to the 
black holes of the transformations Ra,b,c and Ra,b,c- Therefore t G Sa^b,c 
by Proposition 4.3 and hence (5.85) is established. 

Notice that 



(5.90) Ad + [0, b/q) + aZ C Sa,b,c 

by (|5.5 ) and (5.85). Then in the case that Ad 7^ 
(|5.80D follows from ^IT\, §Mi and ([K9l|. 



the conclusion 



(ii) The desired minimality follows from 

(5.91) M.\Sa,b,c = U~ o(^a,v)"([c -co,c + b-co-a) + aZ). 
By Proposition |3.10[ 

U^^o{Ra,b,cT{[c -Co,C + b-Co-a)+aZ)C R\Sa,b,c. 

Then it suffices to prove 

(5.92) R\Sa,b,c C U^^Q{Ra,b,cr{[c - Co, c + 6 - Co - a) + aZ) 
with 

(5.93) {Ra,b,c)^{[c — Co, c + b — cq — a) + aZ) = [cq + a — b, cq) + aZ 
for some nonnegative integer L > 0, c.f. (5.2) and (5.4) in Theorem 



5.4, First we prove that 



(5.94) {Ra,b,c)^{[c - Co, c + 6 - Co - a) + aZ) C [co + a - 6, co) + aZ 

for some nonnegative integer L. By ( 5.6[ ) and finite cardinality of the set 

6Z/gn[0, a), we only need to verify that for any t G [c— Co, c+6— Co— a) + 
aZ) n bZ/q there exists Lq such that {Ra,b,c)^" (t) ^ [cq + a — 6, Co) + aZ. 
Suppose on the contrary that {Ra,b,c)"'(t) ^ [co + a — &, Co) + aZ for 
all n > 0. Then {Ra,b,c)"' (t) ^ [c — Co,c + 6 — Cq — a) + aZ,n > 1, 
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by Proposition 3^, which together with finite cardinahty of the set 
6Z/g n [0, a) imphes the existences of positive integers 1 < Li < L2 
such that {Ra,b,c)^'{t) - {Ra,b,cf^{t) G oL. Set to = {Ra,b,cf'{t). Then 
{Ra,b,c)^(to) and {Ra,b,c)"'(to),n > 1, do not belong to black holes of 
the transformations Rafi,c and Ra,b,c as (-Ra,fe,c)"(^o) - (-Ra,b,c)'^^^'(i) e 
aZ and {Ra,b,c)^{to) - {Ra,b,c)^^~'^{t) e oL where m = n- [n/{L2 - 
Li)\ (L2 - Li), and rn = h- [h/{L2 - Li) J(L2 - Li). Thus to G 5a,fe,c 
by Proposition 3.52, which contradicts to (5.92). 



Next we prove (5.93). By (5.94), it suffices to prove that 

for any distinct ti, t2 G ([c — cq, c + 6 — cq — a) + aZ) fl 6Z/g. Let /ci, ^2 
be minimal nonnegative integers such that 

(i?aAc)'n^l) = (^aAc)''(il) and {RaAcf'it2) = {Ra,b,cf{t2). 

Without loss of generality, we assume that ki < k2. By one-to-one 
correspondence of the transformation Ra,b,c given in Proposition 3.7 
and the selection of integers ki and ^2, 



h = {RaAc)'''{{Ra,b,c)''itl)) e (i?a,v)'^-'^(t2 



aZ, 



which is a contradiction by the assumption ti ^ t2 + aZ (if k2 = ki] 



and the range property of the transformation Ra,b,c in Proposition |3.7 
(if /c2 > ki) . Hence (5.93) is proved. 



By (5.93),M\(u^=o(^aAc)"([c-co,c-co + 6-a)+aZ)) has empty 



intersection with [co+a— 6, co)-|-aZ, the black hole of the transformation 
Ra,b,c- Then by the first conclusion of this lemma, the proof of (5.92) 
reduces to the invariance of the set M\(^u1^^q (Ra^^c)^ {[c — Cq, c — Co + b — 
a) + aZ)) under the transformation Ra^b,c- Suppose, on the contrary, 
that there exists t ^ U^^q (i?a_b,c)"([c — cq, c — cq + b — a) + aZ) such 
that Ra,bAt) e U^^o(^a,6,c)"([c'- Cq, c - Cq + b - a) + aZ). Then t ^ 
[co + a — 6, Co) -f aZ by (5.93), and Ra,b,c(t) 



(Ra.. 



b,c 



(s) for some 

< n < L and s G [c — Co, c — Co-t-6 — a)-|-aZ. It follows that n > 1 from 
the range property of the transformation Ra,b,c in Proposition 3.7 If we 



further select the integer n to be the smallest positive integer such that 
Ra,b,c{i) = {Ra,b,c)"'{s) for somc s G [c — co,c — co + b — a)+aZ. Then t = 
(-Ra,6,c)"~^('S) by the one-to-one correspondence of the transformation 



Ra,b,c given in Proposition 3.7, which contradicts to the assumption 



that t ^ lin=o{Ra,b,cy''i[c - Co, c - Co + 6 - a) + aZ). 



D 



We finish this subsection by the proof of Lemma 5.12 



Proof of Lemma 5.12 . (i) Suppose on the contrary that both [0, b/q) 
and [co, Co + b/q) are not contained in Sa^h^c- Then 0, Cq ^ Sa,b,c by (5.6), 
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which together with (5.6) and Proposition 3.6 imphes that 
(5.95) Sa,b,c C {[b/q, Co + a - 6) U [co + b/q, a)) + oL. 

Thus 



(5.96) 



Ra,b,c{^a,b,c — b/q) C Sa 



b/q 



as Rafi,cit - b/q) = Ra,bAt) - ^/9 fo^ all ^ ^ ^a,b,c C {[b/q, Co + a - 
b) U [co + b/q, a)) + oL. Thus both 5a,fe,c and Sa.b^c — b/q are invariant 
under the transformation -Ra,b,c and have empty intersection with the 



black hole [co + a — 6, cq) +aZ of the transformation -Ra,6,c by (5.96) and 
Proposition 13.81 Hence by the maximality of the set Saf,,c in Lemma 



5.11, Saf,,c — b/q C i5a,f,,c, which contradicts to (5.95) because to — b/q G 
Sa,h,c -~ b/q but to ~ b/q Sa,b,c by (5.6) and ( |5.95 ) where to is the 
smallest positive number in Sa,b,c H [0, a). 

(ii) Suppose on the contrary that both [—b/q, 0) and [cq + a — b — 
b/q, CQ + a — b) are not contained in Sa,b,c- Then a — b/q, Co + a — b — b/q ^ 
Sa,b,c by (5.6). Following the above argument in the proof of the first 
conclusion, Ra,b,c{^a,b,c + b/q) = Sa,b,c + b/q and {Sa,b,c + b/q) H ([co + 
a — b, Co) + aZ) = 0. Hence the set Sa,b,c + b/q is invariant under the 
transformation Ra,b,c and has empty intersection with the black hole 
[co + a — b, Co) + aZ of the transformation Ra,b,c, which contradicts to 
the maximality of the set Sa,b,c established in Lemma |5.11[ 

(iii) Suppose on the contrary that both [0, b/q) and [—b/q, 0) are 
not contained in Sabc- Then 



(5.97) [co, Co + b/q) C Sa,b,c and [cq + a - b - b/q, Cq + a - b) e Sa, 



b,c 



by the first two conclusions of this lemma. First we show that there 
exists a nonnegative integer 1 < D < {2p — q)/{q — p) such that 



(5.98) {Ra,b,c) i[c-co,c + b-co-a) + aZ)n{[co + a-b,co)+aZ) ^ 



Suppose on the contrary that (5.98) does not hold. Then {Ra,b,c)"'i[c — 
Co,c + b — Co — a) + aZ) n {[cq + a — b, Cq) + aZ) = for all < 
n < {2p — q)/{q — p)- Following the argument in (5.29), we have that 



(-Ra,6,c)'^([c— Co,c+6— Co— a)+aZ), <n< {2p—q)/{q—p), are mutually 
disjoint. This together with (-Ra,6,c)"([c — co,c + 6 — co — a) + aZ) = 
{Ra,b,c)''{[c-Co,c+b-Co-a)nbZ/q) + [0,b/q)+aZ,0< n < {2p-q)/{q- 
p), implies that | Uo<n<(2p-g)/(<?-p) (-Ra,6,c)"([c- co, c + 6 - Co - a) + aZ) n 
([0,a)\[co + a-6,co))| = [p/jq- p)\iq - p) /q > \[0,a)\[co + a-b,co)\, 
which is a contradiction. This proves ( |5.98 ). 

By ( |5.98[ ), we may assume that the nonnegative integer D in (5.98) 
is the minimal integer such that (5.98) holds. Following the above 
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argument, we may conclude that 

(5.99) 

{Ra,b,c)"'{[c — co,c + b — co — a) + aZ),0 <n<D, are mutually disjoint. 

Now let us verify the following claim: 



(5.100) {Rafi,cTi[c - Co, c + 6 - Co - a) + aZ) 
for some bn & {0,cl],0 < n < D, and 



(5.101) {Ra,b,c){[c-co,c + b-co 



aZ) 



[bn + a- b, bn) + oL 



[co + a — b, cq) + aZ. 



Proof of Cla ims pJOO^ and ^JO^. If £> = 0, then ( [5l00| ) and ( [5l0l| 
follow from (5.97) and the definition of the nonnegative integer D. Now 
we consider D > 1. Let To = [c — co, c + 6 — cq — a) + oL and define 
T„, 1 < n < D^ inductively by 
(5.102) 

Ra,b,c{Tn-l) if ^ T„„i, 

Ra,b,c{Tn-l) U ([c - Co, c + 6 - Cq - a) + aZ) if G T„_i. 



T 



Clearly Tq = [bo + a — b^b^) + aZ for some 6o ^ (0,a]. Inductively, 
we assume that T^ = [6„, 6„) + aZ for some 6„, &n with 6„ G (0, a] and 
b — a <bn — bn < a. If ^ T„, then either [6„, 6„) C [0, co + a — b) or 
[&„,&„) C [co,a). This implies that 

Tn+l = Ra,b,c{Tn) = [-Ra,b,c(&n), Ra,b,c{bn) + bn — bn) + oL 

(5.103) =: [6„+i,6„+i) + aZ 

for some j>„+i, 6n+i with 6„+i G (0, a] and 6„+i - bn+i = bn - K- If G 
Tn, then 6„ < 0. Moreover bn > Co— a and bn < Co+a— &, as otherwise T„ 
has nonempty intersection with the black hole [cq + a — &, co) + aZ of the 
transformation Ra,b,c, which contradicts to (5.98) and the observation 
that Tn C U!J^^o(-^a,b,c)'"([c — Co, c + 6 — Co — a) + aZ). Therefore 

T„+i = Ra,b,ciTn) ^ {[c - Co, c + b - Co - a) + aZ) 

= [K + [c/b\ b, bn + [c/b\ b + b-a) + aZ 

(5.104) =: [bn+i,bn+i) + aZ 

for some 6„+i, 6„.+i with 6„+i G (0, a] and 6„+i — fe^+i = bn — bn + b — a. 
Combining (5.103) and (5.104) proceeds the inductive proof that 

(5.105) Tn = [bn, bn) + dZ for all < n < D, 

where 6„ G (0, a] and 6„ — 6,„ G [6 — a, a). Observe that {Ra,b,c)^{[c — 
Co,c+b-Co-a) + aZ) C To C U^=o(-Ra,6,c)"([c-Co, c + 6-Co-a)+aZ). 
Then T^ has nonempty intersection with the black hole [co + a — 6, Co) + 
aZ of the transformation Ra,b,c by (5.99). This together with (5.105) 
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implies that either [cQ+a — h—h/q, CQ+a — h) C To, or [cq, Co+b/q) C Td 
or Td = [co + a — b, cq) + oL. Recall that Td fl Sa^y^ = by Proposition 
3.10 Then both [cq + a — & — &/g, cq + a — 6) and [cq, cq + &/g) have 



empty intersection with Td by (5.97). Thus 

(5.106) Ti^ = [co + a - 6, cq) + aZ. 

This together with ( |5.102D , ( |5.103D and ( |5.104D implies that 

(5.107) &n > and 6„ - 6„ = 6 - a for all < n < D. 



The desired conclusions (5.100) and (5.101) then follow. 



D 



Let us return to the proof of the conclusion (iii). By (5.6), (5.97), 
(5.99), (5.100) and Proposition 3.10, either [6„ + a — 6, 6„) C [6/g, cq + 
a — 6) or [6„ + a — 6, 6„) C [cq + bjq, a). This implies that 

(5.108) Ra,hA^n + a-b- b/{2q)) + aZ = bn+i + a-b- b/{2q) + aZ 



for all < n < D - 1. By ( l5.97[ ), ( |5.100[ ), ( |5.101[ ), ( |5.108[ ), and Propo- 
sitions 3^ and 3^, we have that 

{Ra,b,cT{co + a-b- b/{2q)) + aZ 

= {Ra,b,cT{bD + a-b- b/{2q)) + aZ 

= {Ra,b,cT'\bD-i + a-b- b/{2q)) + aZ = • ■ ■ 

= bo-n + a-b- b/{2q) + aZ C SaM,c, <n<D. 

Hence -b/{2q) + aZ = Ra,b,c{c - Co - b/{2q)) + aZ = (i?a,6,c)''+Hco + 
a — b — b/{2q)) + aZ G Sa^b,cj which together with (5.6) implies that 
[—b/q, 0) G iSa^bc- This is a contradiction. D 

5.3. Covering property of maximal invariant sets. In this sub- 
section, we prove Theorems 5^ and 5.7[ 

Proof of Theorem 5j_6. Set Ax := Sa^^c fl [0, Cq -|- a — 6) -|- A -(- aZ and 
Bx := Sa,b,c n [co, a) + A -|- aZ for A G 6Z. We divide the proof into two 
cases. 

Case 1: a/b ^ Q. 



Take to ^ Sa,b,c- Then {Ra,b,c)^{to) ^ Sa,b,c by Proposition 3.8, Write 
{Ra,b,c)"' (to) = to + knb, whcre the strictly increasing sequence {kn}'^=o 
of nonnegative integers is defined inductively by /cq = and 

[c/b\ +1 if to + Kb G [0, Co + a - 6) + aZ 
[c/b\ if to + knb G [cq, a) + aZ 

for n > 0. Then for any nonnegative integer /, 

to + lb = to + knb + (l - kn)b 



(5.109) kn 



+1 



Kn 



(5.110) 



(Ua. 



e[0,(Lc/6J-l)fe]nbZ -OAa 



Bx, U ( Ua, 



6[0,Lc/bJb]n6Z 



AO 
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by (5.109), where kn is so chosen that kn < I < kn+i. Therefore 



{to + lb- [(to + lb)/a\a\ < / G Z} 
C ( UA2e[o,(Lc/6J-i)6]nra Bx^ n [0, a)) U ( Uxjelo,lc/b\b]nbz ^Ai n [0, a)) 



by (3.3) and (5.110). Notice that the lei^ hand side of the above inclu- 



sion is a dense subset of [0, a) by the assumption a/b ^ Q, while its right 
hand side is the union of finitely many intervals that are right-open and 
left-closed by Theorem |5. 4 Thus 



[0,a) = (uii;J-'(5,,,,e + A;6)n[0,a)) 
(5.111) U((5a,fe,, n [0, Co + a - 6) + [c/b\b) n [0, a)) 



and the conclusion (5.17) follows 



(5.6) and the assumption that Sa,b,c 7^ 0- Following the argument in 



Case 2: a/b = p/q and c/b G Z/g for some coprime integers p and 
Take to G Sa^b,c H 6Z/g. The existence of such a point to follows from 



(5.111), we have that 



(5.112) to + lb e ( UAie[o,[c/6j6]nfez ^aJ U ( Ua26[o,(L. 



c/b\-l)b]nbZ 



Bx..) 



for all < / G Z. Observe that {tp, tp + b, ... to + {p-l)b} + aZ = bZ/q. 
The above observation together with (15.112) implies that 



(5.113) bZ/q C ( UAie[o,[c/6jb]nM ^aJ U ( UA2e[o,(Lc/bJ-i)6]nra Bx^)- 



Combining (5.6 ) and (5.113 ) proves the desired covering property ( |5.17 ), 

D 



Proof of Theorem 5. 7 



By Proposition 4.2 and the assumption 



that T>afi^c = 0, we have that 

(5.114) 

t + A ^ Sa,b,c for all t G Sa,b,c n [0, Co + a - 6) and A G [b, [c/b\ b] fl bZ; 

and 

(5.115) t+A ^ 5«,6,c for allt G 5a,fe,cn[co, a) and A G [6, [c/6J6-6]n6Z. 

Therefore the sets Sa^^c fl [0, Co + a — 6) -|- Ai + aZ, Ai G [0, [c/6J 6] fl bZ, 
and iSa^ft^c n [cq, a) -|- A2 + aZ, A2 G [0, [c/b\b — b] D bZ, are mutually 



disjoint. This together with the covering property in Theorem 5.6 and 
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the periodic property (3.3) for the set Safi,c imphes that 



J2 I ^^-'b,c n [0, Co + a - 6) + Ai + aZ) n [0, a) \ 
\ie[o,[c/bib]nbZ 

+ Yl I ^^-'b'- ^ t^O' «) + ^2 + aZ) n [0, a) I 

\2&[0,[c/bib-b]nbl, 

J2 I (^'^^b,c n [0, Co + a - 6) + Ai + aZ) n [Ai, a + Ai^ 
Aie[o,Lc/feJ6]nfez 

+ Yl |(5„,fe,en[co,a) + A2 + aZ)n[A2,a + A2)| 

X2elo,[c/bib-b]nbZ 

( [c/b\ + 1) \Sa,b,c n [0, Co + a - 6) I + [c/b\ \Sa,b,c n [co, a) I 



and hence (5.18) follows. 



(^) Set Aa = {Sa,b,c n [0, Co + a - 6) + A + aZ) n [0, a) and Bx = 
{<Sa,b,c n [co,a) + A + aZ) fl [0,a),A G bZ. By Theorem 5.6, the sets 



Ax,,Xi G [0, [c/6J6] n bZ and 5a2,A2 G [0, [c/b]b - b] f] bZ form a 
covering for the interval [0,a). This together with the assumption 
(5.18) and the periodic property (3.3) for the set Sa^b^c implies that 



a = {lc/b\+l)\Sa,b,cn[0,co + a-b)\+lc/b\\Sa,b,cn[co,a)\ 

Yl i^^ii+ Y i^^^i 

Aie[o,Lc/6J6]nfez A2e[o,Lc/feJ6-b]n6Z 

> I ( UAie[o,[c/6jb]n6Z ^Ai) U ( Llx2e[o,[c/b\b-b]nbz Bx^) \ = a. 

This implies that the intersection of any two of those sets Ax^.Xi G 
[0, [c/b\b] n bZ and -Baj, A2 G [0, [c/b\b - 6] fl 6Z, has zero Lebesgue 
measure. Hence they have empty intersection as those sets are finite 



union of intervals that are left-closed and right-open by Theorems 5^ 
and 5.5 This together with Proposition 4.2 proves that "Dafi^c = 0- D 



5.4. Algebraic property of maximal invariant sets. In this sub- 



section, we prove Theorem 5.8 



Proof of Theorem \5.8[ (i): By ( |2.16 ) and (3.3), we have that 
(5.116) r„,6,e(t + a)= Ya,bAt) + Ya,bAa) for all t G Sa,b,c- 



Now we divide two cases to verify (5.19) for t G [0, a) fl Sa,b,c- 
Case 1: a/b ^ Q. 
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Fo r t G [0,co + a-b)r\ Sa,b,c, we obtain from ( pl3| , dMlj ), ( pl6| ) 
and ( |3.65[ ) that 

ya,b,c{Ra,b,c(t)) = \[0, Ra,b,c(t)) H Sa,b,c\ 

= ^a,fe,c(-Ra,fe,c(0)) + |[-Ra,6,c(0),-Ra,6,c(i^)) H iSa^fo^d 

= Ya,bARa,bAO)) + \Ra,bA[^^t)nSa,b,c)\ 

(5.117) = Ya,bARa,bAO))+Ya,bAt). 

Similarly for t G [cq, a) fl iSa,fe,c, we get 



^a,6,c(-Ra,6,c(^)) 



(5.118) 



\[Ra,bAco), Ra,bAi)) n 5a,fe,c| + >"aAc(Co + [c/^J &) 
|^aAc([Co,i) n Sa,b,c)\ + I [0, Cq + [c/b\b + fl) H 5a,b,,| 

- 1 [co + Lc/&J 6, Co + [c/b\ 6 + a) n ^a,,,,^ | 

|[co, t) n Sa,b,c\ + |[o, Lc/&J& + b) n 5^,^! 

+ |^aAc([0, Co + a - 6)) n Sa,b,c\ - ^aAc(«) 
i;,fe,c(t) + Ya,bARa,bAO)) - Ya,bAa). 



Then ( |5.19D follows from ( |5.117D and (jSJlSj). 

Case 2: a/6 = p/g and c/b G Z/g /or some coprime integers p and 



For every t G [0, Co + a — 6) fl iSa^b^c, we obtain from (3.51 ) and (3.53 ) 
in Proposition 3.8, (5.14) in Theorem 5.5[ and (5.116) that 

Ya,bARa.,bA^)) ~ ^a,fe,c(-Ra,fe,c(0)) + Ya^bA^) 

(5.119) G Ya,bAci + b-a) + Ya,bAt) + Ya,bA(^)Z. 
Similarly for every t G [cq, a) fl Sa,b,c, we have that 

Ya,bARa,bAt)) = Ya,bAt) + Ya,bARa,bA^)) " Ya,bAa) 

(5.120) G Ya,bAci + b-a) + y,,,,,(t) + Ya,bAa)Z. 



Then (5.19) follows from (5.119) and (5.120). 



(ii): Let Ni,N2,6,6' be as in Theorem 5.5, By (5.14), the set ICa,b,c 
of marks is given by 

}Ca,b,c = {Ya,bAiRa,b,cT{c-co + b-a + 5))\0 <n< Ni + N2} + Ya,bA(^)'L. 

This, together with the first conclusion of this theorem and the fact 
that [0, 5) is contained in a gap, implies that 

^a,b,c = YafiARa,bA^) ~^ ^) 

+{nYa,bAci + b-a)\0<n<Ni + N2} + Ya,bAa)^ 
(5.121) = {nYa,bAc, + b^a)\l<n<N, + N2 + l} + Ya,bA(^)Z. 
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On the other hand, it follows from (5.14) and (5.15) that {Ra,b,^ 



\Ni+N2i 



Co + b + S) E 5 + aZ. This together with the first conclusion of this 
theorem implies that 

(5.122) (iVi + N2 + l)yaAc(ci + b-a)e Y^,^{a)Z. 



Combining (5.121) and (5.122) proves that /Ca,fe,c form a finite cyclic 
group generated by ya,fe,c(ci + h - a) + Ya^i.^da)'^- 

(iii) By Theorem 5.4, there exists a nonnegative integer D such 



that {Ra,b,c)^ic — Cq + b — a) + [a — b, 0] + aZ, < n < D, are mutually 
disjoint, and 



(5.123) iRa,b,c) 
Therefore (i?^ 



b,cT{c 



Co + & — a) + [a — &, 0) + aZ = [cq + a — 6, cq) + aZ. 
Co + & — a) G Sa,b,c for all < n < Z), and 



U^=o{^aAc((^a,v)"(c - Co + 6 - a)) + ya,fe,e(a)Z} 

U^=o{{n + l)Fa,6,c(c - Co + 6 - a) + Fa,6,c(a)Z} 



U^±UmF,,6,e(ci + & - a) + ra,6,c(a)Z} 



(5.124) 

where the second equality follows from the first conclusion of this the- 



orem. Moreover, it follows (5.123) that 
(5.125) 



{D + i)raAc(ci + b-a)- ra,M(co) e Ya,bA(^)Z. 



Also we notice that the nonnegative integer D satisfying (5.125) is 



unique as (-Ra,fe,c)"(0) ^ aZ for all positive integers n by the assumption 
a/b ^ Q. This uniqueness, together with (5.124) and (5.125), proves 
the conclusion (iii). D 



Remark 5.13. Let A^^i and N2 be as in Theorem 5.5 Notice that the 



mutually disjoint properties (iv) and (iv)' in Theorem 5.5, nYa^b,cici + 
b — a)^ YafiAa)Z for all < n < A^i + A^2- Thus /Ca,fe,c is a cyclic group 
of order A^i + A^2 + 1- 

6. Maximal Invariant Sets with Irrational 
Time-Frequency Lattice 



In this section, we prove Theorem 2.4 To do so, we need charac- 
terize the non-triviality (2.10) of the maximal invariant set Sa^^c- By 
Theorems 5^ and 5.5[ after performing the holes-removal surgery, the 
maximal invariant set Sa,b,c becomes the real line with marks. This 
suggests that for the case that a/b ^ Q we can expand the line with 
marks by inserting holes [0, b — a) at every location of marks to re- 
cover the maximal invariant set iSa,fe,c- Using the equivalence between 
the application of the piecewise linear transformation Ra,b,c on the set 



Sa,b,c and a rotation on the circle with marks given in Theorem 5.8 
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we can characterize the non-triviahty (2.10) of the maximal invariant 
set Sa,b,c via two nonnegative integer parameters di and d2 for the case 
that a/b ^ Q. 

Theorem 6.1. Let {a,b,c) be a triple of positive numbers satisfying 
a < b < c, [c/b\ > 2,b — a < Co := c — [c/b\ b < a,0 < Ci := c — Cq — 
[(c — co)/a\a < 2a — b, and a/b ^ Q. Then Sa,b,c 7^ ^/ ond only if 
there exist nonnegative integers di and d2 such that 

(6.1) c-{di + l){[c/b\ + l){b - a) - {d2 + l)[c/b\{b - a) e aZ, 

(6.2) [c/b\b+{di + l){b-a) <c< [c/b\b + b - {d2 + l){b - a), 
and 



(6.3) 



where Ea,h,c is defined as in (2.19). 



i^Eafi^c — di, 



The nonnegative integers di and d2 in Theorem 6.1 satisfy (rfi + 
d2 + 1) < a/(b — a) by (6.2), and they are uniquely determined by the 



triple {a,b,c) of positive numbers by (6.1) and the assumptions that 
[c/b\ > 2 and a/b ^ Q. We also notice that the nonnegative integer 
parameters di and c?2 in Theorem |6.1| are indeed the numbers of holes 
contained in [0, Cq + a — b) and [cq, a) respectively. 



In next two subsections, we prove Theorem 6.1, and apply Theorems 



5.7 and 6.1 to prove Theorem 2.4 respectively. 



6.1. Nontrivial maximal invariant sets with irrational time- 



frequency lattices. In this subsection, we prove Theorem 6.1 



Proof of Theorem 6. 1 . {=^) Assume that Sa,b,c 7^ 0- Let D be the 
nonnegative integer in Theorem 5.4 Then {Ra,h,c)^ {[c — cq, c + 6 — cq — 
a) + aZ) = [co + a — 6, cq) + aZ, and the periodic holes (-Ra,6,c)"'([c — 
Co, c + 6 — Co — a) + aZ) = (i?a,6,c)"(c — Co) + [0, fe — a) + aZ, < n < Z), 
are mutually disjoint. This implies that 

(6.4) co + a-6- [c-CQ + di{[c/b\ + l)b + d2[c/b\b) G aZ, 

where di,d2 are the numbers of the indices n & [0, Z) — 1] such that 
(-Ra,6,c)"([c — Co, c + 6 — Co — a) + aZ) is contained in [0, Cq + a — 6) + aZ 
and in [co,a) + aZ respectively. Then (6.1) follows from (6.4). 

Observe that 

D = di + d2 

as for every Q < n < D — 1 the periodic hole (-Ra,b,c)"'([c — Co, c + 6 — 
Co — a) + aZ) is contained either in [0, Cq + a — 6) + aZ or in [co, a) + aZ 



by Theorem 5.4 
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Notice that the periodic holes (/2a,6,c)"'([c — cq, c+6 — cq — a) + aZ), < 



n < di + d2, are mutually disjoint and have length b — ahy Theorem 5^ 
and that there are di (resp. (^2) holes contained in [0, co + a — b) (resp. 
[co,a)) by the definition of integer parameters di and d2- Therefore 
di{b — a) < Co + a — b and d2{b — a) < a — cq, which proves (6.2). 

Let 9a,b,c '■= ya,b,c{^ — Cq + b) bc as in Theorem 5.8, and 9a,b,c = 
^a,6,c(ci). Recall that there are di + d2 + I holes contained in the 



interval [0, a) by Theorem 5.4 This together with (5.116) implies that 

(6.5) Yafifii^-) = a - (di + d2 + l){b - a) 
and 

(6.6) 9a,b,c - yaf>M - Co) G Ya^i„c{(^)Z, 

where the inclusion holds because [c — Cq, c + b — Cq — a) is a black hole 
of the transformation Ra,b,c, and 

(6.7) Ya^hA'^ - Co) - 9a,b,c = ->^aAc(a). 



From Theorem 5.8, we see that the marks are located at Ya^b,c{'^ ~ 
Co) + n9a,b,c + (a — {di + d2 + l){b — a))Z, < n < di + d2. This 
together with ( 6.5[ ), (6.6) and (6.7) implies that the locations of marks 
are n9a,b,c + (ct — (c^i + d2 + l){b — a))Z, 1 < n < di + d2 + 1. 

Recall that there are di holes contained in [0, Co + a — b). Then 
^a,6,c(co + a — b) = Cq + a — b ~ di{b — a) , which implies that 

(6.8) Co-{d^ + l)ib-a)-{di+d2 + l)9a,b,ce {a-{di + d2 + l){b-a))Z. 

Let m be the number of holes (-Ra,fe,c)"([c — co, c + 6 — co — a) + aZ) = 
(-Ra,6,c)"'([ci, ci + b — a) +aZ), <n < di + d2, contained in [0, ci) + aZ. 
Due to the one-to-one correspondence between holes and marks, m is 
also the cardinality of the set {l < n < rfi + ^2 + l\n9afi,c G [0, 9a^bA + 
(a — (c?! + (i2 + 1)(& — a))Z}. This, together with the observation that 
[ci , Ci + 6 — a) is the black hole of the transformation Ra,b,c, implies that 

(6.9) 9a,b,c = ci-m{b-a). 

Let m be the unique integer such that {di + ^2 + ^)9a,b,c G fh{a — {di + 
(^2 + 1)(& — a)) + [0, a — {di + (^2 + 1)(& — a))- We want to prove that 

(6.10) m = m. 

For any 1 < I < rh, there exists one and only one 1 < ni < di + d2 + 1 
such that ni9a,b,c G l{a — {di + d2 + l)(b — a)) + [0, 9a,b,c), which implies 
that rfi < m. Now we prove that m < rh. Suppose on the contrary 
that m > m. Then there exists an integer l<n<di + d2 + l such that 
n9a,b,c e [0,9aM) + i^-idi + d2 + l){b-a)){Z\{l, . . . ,m}). This implies 
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that n9a^b,c > (?Ti + l)(a— {di + 62 + 1)(6 — a)), which is a contradiction 
as 6a,h,c < n6a,b,c < {di + d2 + l)6a,b,c < {m + l){a-{di + d2 + l){b-a)) 



by the definition of the integer m, and hence (6.10) is estabhshed. 
From (Q, Q and ( [610| , 

{di + d2 + l)(ci - ■m{b - a)) = {di + ^2 + l)6'a,6,c 
= Co — {di + 1)(6 — a) + m(a — (rfi + ^2 + 1)(& — a)), 
which imphes that 
(6.11) ma = {di + ^2 + l)ci - Cq + (rfi + 1)(6 - a). 



Then the condition (6.3) follows from (6.8), (6.9) and (6.11), and the 



definition of the integer di. 

{•^=) Let Ci = c — Co — [(c — Co)/aJa, and let di and d2 be as in 
(lO and (Q. Then ci ^ by a/b ^ Q, and -a < -Cq + b - a < 



{di + d2 + l)ci-Co + {di + l){b-a) < {di + d2 + l)ci < {di + d2 + l)ahy 
(6.1) and (6.2). Also from ( |6.1 ) and (6.2), we see that (di + ^2 + l)ci — 
co + {di + l){b-a) e {di + d2 + l)[c/b\b-c+[c/b\b+{di + l)b+aZ = aZ. 
Thus 

(6.12) (rfi + ds + l)ci - Cq + {di + l){b -a) =ma 
for some integer < m < di + d2- Set 

(6.13) 9a,b,c = ci-m{b-a). 
Then 

(6.14) (rfi + rf2 + l)^aAc = Co-{di + l){b-a)+m{a-{di+d2 + l)ib-a)) 



by (6.12). This together with < m < di + d2 and < cq — (rfi + 
1)(6 — a) < a — ((ii + ^2 + 1)(^ — c^) implies that 

(6.15) ea,b,c e (0, a - (c/i + d2 + l)ib- a)). 
We claim that 

(6.16) {n - n)ea,b,c ^ {a - (rfi + ^2 + l){b - a))Z 

for all 1 < n 7^ n' < (ii + (i2 + 1. Suppose on the contrary that (6.16) 



are not true. Then k6a,b,c = l{a — {di + d2 + l){b — a)) for some integers 
/ G Z and fc e [1, di + i] H Z. Then k{m - [c/b}) = l{di + ^2 + 1) and 
k{m — [( [c/b\ b/a)\ ) = l{di + d2 + 2) by the assumption a/b ^ Q. Thus 
/ = k{[_{[_c/b\b/a)\ — [c/6J), which is a contradiction as 1 < / < /c by 



(6.15). 



Denote /C^ 



6,c 



hrf2 + l 



(di + ^2 + 1)(&- a))Z and 
rewrite ]Ca,b,c as {;z„}^LV^^ + (a - {di + 4 + 1)(6 - a))Z where < 
Zi < Z2 < ■ ■ ■ < Zdi+d2+i < a — {di + d2 + l){b — a) . The existence of 
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such an increasing sequence {zn} 



di+d2+l 
n=l 



6 G (0, cq — ((ii + l)(6 — a)) (respectively 5 G {co — {di + 



follows from (6.16). Given any 



){b — a), a — {di + 



(^2 + 1)(& — a))), it follows from (6.14) and (6.15) that for any integer 
k G [0, m] (resp. k E [0,171 — 1]) there is one and only one integer n G 
[1, di + d2 + l] such that n6a,h,c € k{a — {di + d2 + V){h — a)) + [5, 5 + 0a,b,c) 
and for k G Z\[0, m\ (resp. k G Z\[0, r/i— 1]), and there is no integer n ^ 
[1, di + d2 + l\ such that n6a,h,c G k{a— {di + d2 + \){h — a)) + [5, 5 + 6a,h,c)- 
The above observations together with (6.15) and (6.16) imply that 
(6.17) 
i^{[6,^+~ea,b,c)n{{zk]tX^'^' + {0,a-{di + d2 + l){h-a)}) =m + l 

for 5 G (0,Co — {di + \){h — a)), and 

(6.18) i^{[6,6 + eaM)^{{^k}tJ'i''''^' + {0,a-{di+d2 + l){b-a)}) = m 



for (5 G (co - (rfi + 1)(6 - a), a - (rfi + ^2 + 1)(& - a))- 



Now let us expand marks located at {zi} 



di+d2 + l 
1=1 



{di + d2 + 



l){b - a))Z to holes of length b-a located at {yi}^^'^^^^ + oL on the 
real line by 

(6.19) y^ = zi^{l- 1)(6 - a), 1 < / < c?i + ^2 + 1. 
Clearly < yi < 1/2 < • • • < ydi+d2+i < ^t- Now let us prove 

(6.20) {Ra,b,cT{c-Co) + aZ = yi(^n) + a^ for all < n < rfi + ^2, 

by induction on < n < rfi + ^2, where l{n) G [l,di + d2 + 1] is the 
unique integer such that z;(„) G (?7, + l)6'a,6,c + (a — (rfi + 'i2 + l)(^ — a))^- 
Applying (6.17) with sufficient ly s mall 6 and recalling ^i > proves 



that Zm+i = Oabc- Combining ( |6.3[ ) and (6.14) gives 

(6.21) 

Zd^+i = Co+a-b-di{b-a) = {di+d2+l)0a,b,c-'m{a-{di+d2+l){b-a)). 



Thus 

(6.22) 

and 

(6.23) 



ym = ym+i = Zm+i + m{b - a) = dafi,c + m{b - a) 



yi(di+d2) = Udi+i = Zd^+i + di(b- a) = CQ + a-b. 



C\ 



The conclusion (6.20) for n = follows from (6.22). Inductively we 
assume that (6.20) holds forn = fc < t/i + c?2 — 1- Then 2;;(fc) 7^ Cq — ((ii + 
1)(6 — a) by (6.14), (6.16) and the observation that /(A;) 7^ (ii + ^2 + 1- 
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(i?aAc)'+'(c-Co) 



b by (|6.23|) and 

+ aZ 



If zi^k) < Co- {di + 1)(6 - a), then yi(^k) < Cq + c 

= Ra,b,c{{Ra,b,c) {C — Cq)) E Ra,b,c{yi{k) 

-- yi{k) + [c/h\ b + b + aZ 

-- zii^k) + 0a,b,c + {'m + l{k)){b- a) + aL. 



(6.24) 



Note that ^^(fc+i) = zi{^k) + Oa,b,c or zi{^k+i) = Zk(i) + 9a,b,c -{a- (rfi + d2 + 
l)(fe — a)). For the first case, l{k + 1) = l{k) + m + 1 as [-2/(a;), zuk^ 



OaM) n ({^fcjti + {0, a - (cii + ^2 + !)(& - a)}) = m + 1 by (|6.17D 



and hence 

{Ra,b,c)''^\c ~ Co) G Zi(fc) + ^a,b,c + (m + /(A:))(6-a) + aZ 

(6.25) = z;(fc+i) + (/(fc + 1) - l)(6-a) + aZ = ?//(fc+i) + aZ. 

Similarly for the second case, l{k + 1) = l{k) + m + 1 — {di + d2 + 1) 
since #([0, z,(fc+i)) + (a - {d, +d2 + l){b- a)) n ({^Jtf '"■' + {0, a - 
(di + d2 + !)(& - a)})) = #(([0,z;(fc)) n {^fcjtf '^'}) U (W)'^W + 



^. 



n{2;fc} 



di+d2+l 



})) = /(A;) _ 1 + rn + 1 = m + l{k) by (6.17). Thus 



(i?a. 



\k+li 



b,c) (c-co) e 2;;(fe) + 6'a,5,c + ("^ + ^(/i;))(&-a) + aZ 
= zi^k+i) + (a - (di + d2 + l){b - a)) 

+ {l{k + 1) + (di + d2 + l)- 1)(6 - a) + aZ 
(6.26) = yi{k+i) + aZ- 

This shows that the inductive conclusion holds when zk^^.^ < cq — {di + 
1)(6 — a). Similarly we can show that the inductive conclusion holds 
when zi( k) > c p - (rfi + 1)(6 - a). 

From (6.20), we see that for any < n < di + d2 — I, (-Ra,6,c)"([c — 
Co, c — Co + b — a)) + aZ = [yi(n),yi{n) + b — a) + aZ is contained either 
in [0, Co + a — 6) + aZ or [cq, a) + aZ, and {Ra,b,c)^{[c — co,c — co + b — 
a)) + aZ = [co + a — b, cq) + aZ. Therefore Safi,c is the complement of 
^ri=o ^{[yi{n)-,yi(n) + b — o) + oL) and hence it is not an empty set. D 



6.2. Proof of Theorem |2]4[ (XII): We observe that Q{x[p,c), oL x 
Z/6) is not a Gabor frame if and only if T>af,fi 7^ if and only if Sa^j^c 7^ 



and (5.18) does not hold if and only if the triple (a, 6, c) satisfies (6.1) 



(Q,^6^,andc-(Lc/6J + l)(di + l)(6-a)-[c/6J((ii + l)(6-a) ^ a. 
In the above argument, the first equivalence holds by Theorem |3.2[ the 
second one follows from (|3.27[ ) and Theorem 5.7, and the last one is 
obtained from Theorem 6.1 and the observation that (5.18) holds if 
and only if c — ( [c/6J + 1) (t/i + 1) (6 — a) — [c/6J (^2 + 1) (& — a) = a as 
there are d^ holes of length 6 — a in 5a,f,,c n [0, Cq + a — 6) and ^2 holes 
of length 6 — a in Saft.c H [cq, a) by Theorem 5.4 
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7. Maximal Invariant Sets with Rational Time-Frequency 

Lattice 



In this section, we prove Theorem |2.5[ To do so, let us consider 
how to expand the hne with marks for the case that a/h = p/q for 
some coprime integers p and q and c/b G Z/g. Unhke for the case 
that b/a ^ Q, we insert gaps of either large or small sizes at each 
location of marks and also insert a gap at the origin, which makes the 
augmentation operation rather delicate and complicated. But on the 
other hand, we get the help from the finite cyclic group structure for 
the marks stated in Theorem 15.81 

Theorem 7.1. Let (a, b, c) be a triple of positive numbers satisfying a < 
b < c,b- a < Co := c- [c/b\b < a,0 < Ci := [c/b\b - l{lc/b\b/a)\a < 
2a — b, [c/b\ > 2, a/b = p/q for some coprime integers p and q, and 
c/b G Z/q. Then Sa,b,c i^ ^ if cmd only if the triple {a,b,c) of positive 
numbers is one of the following three types: 

1) Co < gcd(a,ci). 

2) b- Cq < gcd(a, Ci + b). 

3) There exist nonnegative integers di,d2-,d^,di such that 

(7.1) < 5rf := a - (rfi + ^2 + 1)(& - a) G Nbl^/q, 

(7.2) Nci + {di + d^ + l){b-a) eaZ, 

(7.3) (c/i + ^2 + l)(A^Ci + (rfi + 4 + 1)(6 - a)) - (d^ + ds + l)a G NaZ, 

(7.4) Co = {di + 1) (6 - a) + (rfi + 4 + l)Bd/N + S 

for some 5 G (— m.m{Bd/N, a — Co),m.m{Bd/N, cq + b — a)), 

(7.5) gcd(A^Ci + (rfi + (ia + 1)(6 - a), Na) = a, 
and 

(7.6) #<., = di. 



where N = di + d2 + d3 + d4 + 2 and -Effec ^■^ defined as in (2.20). 



In Theorem 7.1, we insert a gap of large size at the origin for the first 
two cases, while a gap of small size is inserted at the origin for the third 
case. For the first two cases, no gaps of small size have been inserted 
at any location of marks and the size of gaps inserted is always co for 
the first case and b — cq for the second case. For the third case, the 
nonnegative integer parameters di,d2 are indeed the numbers of gaps 
of size b — a + \6\ inserted in [0, Co + a — 6) and [co, a) respectively, and 
the nonnegative integer parameters ^3,^4 are the numbers of gaps of 
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size \6\ inserted in [0, cq + a — 6) and [cq, a), excluding the one inserted 
at the origin, respectively. 



In next two subsections, we give the proofs of Theorems 7.1 and 2.5 



7.1. Nontrivial maximal invariant sets with rational time-frequency 
lattices. In this subsection, we prove Theorem 7.1 The necessity of 



Theorem 7.1 follows essentially from Theorems 5.5| and [5l8 We exam- 
ine five cases to verify the sufficiency. For the case 1) cq < gcd(ci,a), 
we show that [cq, gcd(ci, a)) + gcd(ci, a)Z is an invariant set under the 
transformation Ra,b,c and it has empty intersection with black holes of 
transformations Ra,b,c and Ra,b,c- This together with the maximality of 
the set Sa,b,c in Lemma 5.11 implies that Sa^^c 7^ 0- Similarly for the 



case 2) 6 — Co < gcd(a, ci + 6), we verify that [0,gcd(a, ci + b) — b + co) + 
gcd(a, Ci + 6)Z is invariant under the transformation Ra,b,c and it has 
empty intersection with black holes of transformations Ra,b,c and Ra.b,c- 
For the case 3), we start from putting marks at /iZ and insert gaps 
^=^(sgn((5 + b/{2q)) + [-1, 1]) at marks located at Imh + NhZ, 1 < 

I < di + d2 + I, and y (sgn((5 + b/{2q)) + [—1, 1]) at other marked loca- 
tions, where A^ = di+d2+d3+d4+2, h = {a-{di+d2 + l){b-a))/N-\6\ 
and m = {Nci + {di + d^ + l){b — a)) /a. We then show that the gaps 
just inserted form a set that is invariant under the transformation Ra,b,c 
and that contains black holes of the transformations Rabc and Ra b c- 



Proof of Theorem 7. 1 . 



Let Ni, N2, 5, S', h be as in Theorem 5.5 



Case 1: 6 = Cq + a — b and 6' = 0. 

In this case, A^2 = and (-Ra,6,c)"([ci, Ci + Cq) + aZ), < n < Ni, are 
mutually disjoint gap with {Ra,b,c)^^{[ci,Ci + Cq) + aZ) = [0,Co) + aZ 



by Theorem 5.5 Therefore A^^i 7^ as Ci > 0. Observe that 

iRa,b,cTi[ci, ci + Co) + aZ) = [ci, ci + Co) + n{ci - a) + aZ, < n < A^i 

because —a < ci — a < and {Ra,b,c)"'{[ci, ci + co) + aZ) C [cq, a) + aZ 
for all < n < A^^i — 1. Replacing n by Ni in the above equality and 
recalling that (-Ra,b,c)^^([ci, ci + co) + aZ) = [0, co) + aZ gives 



ci + A''i(ci — a) & aZ, 



which implies that 

(7.7) 



Cl 



-A''i(ci — a) + ka 



for some integer k. Write Ci/b = r/q and let m = gcd(p, r). Then 
it follows from (7.7) that A^^i + 1 G pZ/m. This together with mutual 
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disjointness of the gaps [ci,ci + co)+n(ci — a) + aZ, < n < Ni, implies 
that 

(7.8) A^i + 1 = p/m, 

as otherwise p/m < Ni and [ci, ci + co) + aZ = [ci, ci + cq) + {p/m){ci — 
a) + aZ. Observe that 

U„iQ(n(ci— a)+aZ) = U„J^ (n(ci— a)+aZ) = {0,mb/q, . . . , (p— m)6/g}+aZ. 

Therefore the mutual disjointness of the gaps [ci,Ci + Cq) + n(ci — 
a) + aZ, < ra < A^i, becomes cq < mfe/g = gcd(ci, a). We notice that 
U^io[ci,Ci + Co) + n{ci-a) + aZ = [Jl^^o'\[0,mb/q)+mnb/q + aZ) =R 
if Co = mb/q. This proves the desired first condition cq < gcd(ci, a) in 
Theorem I7.1[ 



Case 2: S = and S' = Cq — a. 

In this case, A^2 = and (-Ra,6,c)"([ci + cq — a, ci + b — a) + aZ), < 
n < Ni, are mutually disjoint gap with {Ra,b,c)^^ {[ci + cq — a,ci + b — 



a) + aZ) = [cq + a — 6, a) + aZ by Theorem 5.5 Therefore A?"! > 1 as 
Ci < 2a — 6. Observe that 

(-Ra,6,c)'^([ci+Co— a, Ci+6— a)+aZ) = [ci+Cq— a, Ci+6— a)+n(ci+6— a)+aZ 

for all < n < Ni, because 0<ci + b — a<a and (-Ra,6,c)"([ci + cq — 
a, ci + 6 — a) + aZ) C [0,co + a — 6) + aZ for all < n < iVi — 1. Replacing 
n by A''i in the above equality and recalling that {Ra,b,c)'^^ {[ci + cq — 
a,Ci + b — a) + aZ) = [cq + a — b,a) + oL give 

(7.9) (A^i + l)(ci + 6-a) GaZ. 

This together with mutual disjointness of \c\ + cq — a, ci + 6 — a) + 
n(ci + 6 — a) + aZ, < n < A?"!, implies that 

A/"! + 1 = a/gcd(ci + 6, a), 

as otherwise a/gcd(ci + 6, a) < A''i and \c\ + cq — a, ci + & — a) + 
aZ = [ci + Co — a, Ci + 6 — a) + (a/gcd(ci + 6, a))(ci + 6 — a) + aZ, 
which is a contradiction. Therefore mutual disjointness of the gaps 
(-Ra,b,c)'^([ci + Cq — a, Ci + 6 — a) + oZ), < n < Ni, becomes mutual 
disjointness of the gaps [ci + Cq — a, Ci + 6 — a) + zgcd(ci + b,a) + aZ, < 
i < a/gcd(ci + 6, a) — 1, which holds if and only if 6 — cq < gcd(ci + 6, a). 
Also we notice that 

U^io(^aAc)"([ci + Co - a, ci + 6 - a) + aZ) 
= U%^Q ([ci + Co — a, ci + 6 — a) + igcd(ci + b,a) + oL) 

= ci + 6 - a + [-gcd(ci + 6, a), 0) + gcd(ci + b, a)Z = R 
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if 6 — Co = gcd(ci + b,a), which contradicts to Sa,b,c 7^ 0- This leads to 



the desired second condition b — cq < gcd(ci + 6, a) in Theorem 7.1 



Case 3: < S < Co + a — b and 5' = 0. 



By Theorem 5^ , iVi > and N2 > 1. Denote by di , ^2 the number 
of big gaps {Ra,b,cT{c - cq + [0, 6 - a + 5)), < n < A^i - 1, of length 
b — a + 6 contained in [0,co + a — b — 6) + oL and in [cq, a) + oL 
respectively, and similarly denote by (^3 and d/^ the number of small 
gaps (i?a,fe,c)'"([co + \clb\b- 6, Co + [c/b\b)), 0<m<N2-l,oi length 
6 contained in [6,Co + a — b — 6) + aZ and in [co, a) + oL respectively. 



Now let us verify that (7.1 )-(7.6) hold for the above nonnegative integer 
parameters o?i, (^2, c?3, ^4. 



Proof of (7. 1 ). By Theorem 5.5 , for every < n < A^i — 1, the gap 
{Ra,b,c)^{c—co+[0,b—a+6)+aZ) is either contained in [0,co+a—b)+aZ 
or [co, a) + aZ. Hence 

(7.10) Ni = di + d2. 
Similarly 

(7.11) N2-l = d^ + di 

as for any < m < A^2 — 1, the gap (i?a,6,c)™'([co + \c/b\b — (5, cq + 
[clb\b)) = {RaAcr+\[co + a-b-6,Co)\[co + a- b, Cq) + aZ) of length 
5 is contained either in [0, Cq + a — b — 6) + aZ and in [co,a) + aZ. 
Combining (^^, ( |5.14D , ( |5.15D , ( |7.10D , and ( |7.1lD , we obtain that there 
are (di + 0^2 + 1) gaps of length b — a + 6 and (^3 + ^4 + 1) gaps of length 
6, and A^ := [di + ^2 + c?3 + ^4 + 2) intervals of length h on one period 
[0,a). Therefore 



(7.12) 



< a- (^1 + ^2 + l){b 



N{h + 5)e NbZ/q. 



This proves (7.1). 



Proof of ( 7.2). By (5.10), (5.12) and the definition of nonnegative 



integers (ij, 1 < i < 4, we obtain that 

(7.13) c - Cq + b - a + 5 + di{[c/b\ +l)b + d2[c/b\b G cq + aZ, 
and 

(7.14) Co + [c/b\b -5 + d^{[c/b\ + 1)6 + di[c/b\b G aZ. 



Adding ( |7.13D and ( |7.14D leads to c + {di + 4 + l){[c/b\ + 1)6 + (c?2 + 
d^) \c/b\b G Co + aZ. Then A^ci + (t/i + 4 + 1)(6 - a) G aZ and ( [f^ 
is true. 



Proof of (7. 4)- By Theorem 5.5 and the definition of the integers 
di and d^, the interval [0, Cq + a — 6 — 5) is covered by di gaps of length 
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b — a + 6, (is + 1 gaps of length 6, and di + d^ + l intervals of length h. 
This together with ( 7.12[ ) leads to 

Co + a-b-6 = di{b-a + 6) + {d3 + l)6+{di + d3 + l)h 
(7.15) = di{b-a) + {di + d3 + l)Bd/N. 



This proves (7.4). 

Proof of (7.3). Substituting the expression in (7.15) into (7.13), 
we obtain that 

aZ 3 c — Co + b — a + S — Co + di{[c/b\ + l)b + d2[c/b\b — dia 
= di{[c/b\ +I)b + d2[c/b\b+ [c/b\b + b - {di + l)a 
-(rfi + 1)(6 - a) - (rfi + 4 + l)Bd/N 
(7.16) = {di + d2 + l)[c/b\b-{di + ds + l)Bd/N. 

Multiplying A^ at both sides of the above equation leads to the desired 
inclusion (7.3). 



Proof of (7.5). Define 



(7.17) 



m = {Nci + (rfi + 4 + 1)(6 - a)) /a. 



Then m is a positive integer in [1, A^ — 1] by (7.2) and the observation 
that < Nci + {di + d3 + l){b-a) < N{2a-b) + {di + d3 + l){b- a) = 
Na — ((^2 + (^4 + 1)(6 — a) < Na. Let Ya^b,c be as in Theorem 5.8 and let 
mi be the nonnegative integer in [0, A^ — 1] such that Ya^b,c{^i + b — a) G 
rriih + Ya^b^c{ci)Z. We claim the following: 

(7.18) mi = m. 

Recall that {Ra,b,c)^^{[ci, ci + b — a + 6)+ aZ) = [co + a — b — S, co) + aZ 
by Theorem 5^, and that there are di + d^ + 1 gaps in the interval 
[0,co + a — b — 5). This together with Theorem 5.8| that 

(7.19) (rfi + da + l)mih - {d^ + 4 + l)h G Ya,bA(^)^ = ^^Z. 

Then the number of gaps of length b — a + 6 contained [0, ci) is {{di + 
d2 + l)mih - {di + d3 + l)h)/Ya,b,c{a) = ((rfi + d2 + l)mi - {di + rfa + 
1))/N. This implies that there are mi gaps contained in [0,ci) with 
{{di + d2 + l)mi — {di + d3 + l))/N of them are gaps of length b — a + 6. 
Hence 

{di + d2 + l)mi - {di + ^3 + 1) 



Cl 



mih + {nil 



N 



-)^ 



{di + d2 + l)mi - {di + 4 + 1) 
+ — [b -a + S) 

= {mia-{di + d3 + l){b-a))/N. 
This together with ( |7.17[ ) proves (7.18). 
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We return to the proof of (7.5). The above claim (7.18), together 



with (7.10), (7.11), and the observation that lCa,b,c is a cychc group 
generated by Ya,b,c{ci + h — a) and has order Ni + N2 + 1 = N hj 



Theorem 5.8 and Remark 5.13, proves (7.5). 

Proof of (7.6). Applying ( |7.17[ ) and (7.18), and recalling that [ci,ci+ 
6 — a) is a hole in the complement of the maximal invariant set Safi,c-i 
we have that 



(7.20) 



Ya^bM) -mhe NhZ. 



Then n G -E'ffcc i^ ^^^ ^^^y i^ {Ra,b,c)"'[ci, Ci + b — a + S) is a big gap 
contained in [0, co + a — b) + al^. This implies that the cardinality of the 
set -E^ftc is equal to di from the definition of the nonnegative integer 
d,. 

This completes the proof of the necessity for the case that 6 G (0, co + 
a — b) and 6' = 0. 

Case 4: 5' G (cq — a, 0) and 5 = 0. 



By Theorem 5.5, A^^i > and N2 > 1. Denote by di, ^2 the numbers 
of big gaps {Ra,b,c)"'{c — Co + [5' ,b — a)),0 < n < Ni — 1, oi length 
b — a — 6' contained in [0, cq + a — b) + oL and in [cq — (5', a + 5') + oL 
respectively, and similarly denote by d^ and ^4 the numbers of small 
gaps (i?a,fe,c)"([co + \clb\b, Co + \clb\b-b')), < m < A^s - 1, of length 
—5' contained in [0, co+a— 6)+aZ and in [cq— 5', a+5')+aZ respectively. 
We may follow the argument for the first case and prove the desired 
properties (7.1)-(7.6) with the above nonnegative integers o?i, o?2, 0^3 and 
d^. 

Case 5: 5 = 5' = 0. 

By Theorem |5.5[ A^i > and A''2 > 1. Denote by di, d^ the numbers 
of gaps (-Ra,6,c)"(c — Co + [0, 6 — a)), < n < Ni — 1, of length b — a 
contained in [0, Cq + a — 6) + aZ and in [cq, a) + aZ respectively, and 
similarly denote by ds and d^ the numbers of (-Ra,fe,c)™'(co), I <m < N2, 
contained in (0, cq + o — 6) + aZ or [cq, a) + aZ respectively. Then we may 
follow the argument for the first case line by line and establish (7.1)- 
(7.6) with the above nonnegative integer parameters di,d2,d^ and 1^4. 

This completes the proof of the necessity for the case that 5 = 5' = 
and also the proof of the necessity. 



{•^=) We examine five cases to prove the sufficiency. 
Case 1: co < gcd(ci,a). 
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Let A^ + 1 = a/gcd(ci, a) for some nonnegative integer N and define 
T = ( U^^o [co, gcd(ci, a)) + n{a - Ci)) + aZ. Then 

T = (U^o [co,gcd(ci,a)) + zgcd(ci,a)) +aZ 
(7.21) = [co,gcd(ci,a)) +gcd(ci,a)Z, 

and T has empty intersection with black holes of the transformations 
Ra,b,c and Ra,b,c, since T fl [0, cq) = T fl [ci, cq + ci) = 0, and for any 
teT, 

Ra,b,c(t) = t + ci e [co, gcd(ci, a)) + ci + gcd(ci, a)Z = T. 

Therefore T C iSa^b^c (in fact T = 5a,6,c) as Safi,c is the maximal invari- 
ant set that has empty intersection with the black hole of the trans- 
formation Ra,b,c by Lemma 5.11, Thus Sa,b,c is not an empty set as 



the restriction of the set T on [0, a) consists of a/gcd(ci, a) intervals of 
length gcd(ci, a) — Cq > 0. 

Case 2: 6 — cq < gcd(a, ci + h). 

Write a/gcd(a, Ci + 6) = A^ + 1 for some nonnegative integer A^ 
and define T = ( U^q [0,gcd(a, Ci + b) - b + Cq) + i{ci + b - a)) + 
oL = [0, gcd(a, Ci + b) — b + Cq) + gcd(a, Ci -|- 6)Z. Then T' has empty 
intersection with black holes of the transformations Ra,b,c and Ra,b,c, 
since T' fl [ci, ci + 6 — a) = T' fl [cq + a — &, a) = 0, and for any t E T, 

RafiAt) = t + ci + be{[0,gcd{a,ci + b)-b + co) 
+ci + b + gcd(a, ci + 6)Z = T'. 

Therefore T' C Sa,b,c (in fact T' = Sa^b,c) as 5a^{,^c is the maximal in- 



variant set that has empty intersection w ith t 
transformations Ra,b,c and Ra,b,c by Lemma 5.11 



le black holes of the 
Thus Sabc is not an 



empty set as the restriction of the set T' on [0,a) consists of A^ + 1 
intervals of length gcd(a, ci + b) — b + co > 0. 

Case 3: There exist nonnegative integers di,d2,d3,di and 6 G 



(0, mm{Bd/N, Cq + a - b)) satisfying ([7J|)-([7i3|) 
In this case, we set 



(7.22) 



h={a-{di + d2 + l){b - a))/N - 6, 



(7.23) m = {Nci + {di + d3 + l){b-a))/a, 
and 

(7.24) m = {{di + d2 + l)m - (di + 4 + 1))/A^. 
Then 

(7.25) 0<hE bZ/q 
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by (7.1) and (7.4); m is a positive integer no larger than N — 1, i.e., 

m G Zn [1,A^-1] 

b) + {di + ds + l){b- a)) /a < N; and m 



(7.26) 

as < Nci/a < m< {N{2a 

is a nonnegative integer no larger than m, 



(7.27) 



m e [0, m] n Z 



by (7.3). Moreover, 



(7.28) 



m 



m 

N 



a - 


- {di + d2 + l){b-a) 


-.a - 


N 

di + rfg + l,, , 



m{b 



Ci. 



In order to expand the real line M with marks at hZ to create an 
invariant set under the transformation Ra,b,c, we insert gaps [0, b — a+S) 
located at Imh + NhZ, 1 < I < di + d2 + I, and gaps [0, S) otherwise. 
Recall from ^^ that (/ - l')mh ^ NhZ for all 1 < / ^ /' < A^. 
Therefore we have inserted di + d2 + l gaps [0,b — a + 6) and d^ + d^ + l 
gaps [0,(5) on the interval [0,Nh). Thus after performing the above 
expansion, the interval [0, Nh) with marks on [0, Nh) fl hZ becomes 
the interval [0, Nh + {di + d2 + l){b - a + 6) + (4 + ^4 + 1)6) = [0, a) 
with gaps [yi, yt + hi),0 < i < N - 1, where = yo < yi < . . . < yo 
and hi E {b — a + 6,6},0 < i < N — 1. Now we want to prove that 

(7.29) y^ = ci. 

For that purpose, we need the following claim: 

Claim 7.2. For s E [0, A^ — 1] fl Z, the cardinality of the set {I E 
[1, di + d2 + 1] r\ Z| Imh E sh + [0,mh) + NhZ} is equal to rh + 1 if 
1 < s < di + ds + 1, and rh otherwise. 

Proof. For any i E Z, let ki = \_{iN + m + s — l)/m\ the unique integer 
such that kimh E [sh, sh + mh) + iNh. Therefore 1 < fcj < di + ^2 + 1 
if and only if m < iN + m + s — 1 < (di + ^2 + l)m + m — 1 if and only 
if 1 - s < zA^ < {di + d2 + l)m -s = Nrh + (di + rfg + 1 - s). Therefore 

#{/ G [1, c^i + ^2 + 1] n Z| Imh E sh+ [0, mh) + NhZ} 
= ^ #{/ e [1, di + d2 + l]n Z\ Imh E sh+ [0, mh) + iNh} 

= J2i^{kiE[l,di + d2 + l]nZ} 

iez 
= #([(1 - s)/N, m + (rfi + 4 + 1 - s)/N] n Z). 

Counting the number of integers in the interval [(1 — s)/N, m + {di + 
(is + 1 — s)/N] proves the claim. D 
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We return to the proof of the equahty (7.29). By Claim 7.2, we have 
inserted m interval of length h — a + 5 and m — m interval of length 6 
in the marked interval [0,mh). So after performing the expansion, the 
mark located at mh on the line becomes the gap located at mh + {m — 



m)6 + m(6 — a + 5), which is equal to ci by (7.28). This completes the 
proof of the equality (7.29). 

Next we show that 



(7.30) 



yd^+d-i+i = co + a-b~ 6. 



By (7.6), we have inserted di gaps of length b—a+S and {di+d3 + l) — di 
intervals of length S in the marked interval [0, {di + d^ + l)h). Therefore 
the mark located at {di + d^ + l)h becomes {di + d^ + l)h + di{b — 
a + 6) + {ds + 1)6 = Co + a — b — 6 after inserting gaps, where the last 
equality follows from ( |7.4 ). Hence (7.30) follows. 

Then we prove by induction onO<k<N — 1 that 



(7.31) iRa,b,c)Hc - Co) + aZ = yii^k) + aZ, < k < d^ + d 



2, 



and 

(7.32) 

{Ra,b,cr{co + a-b-6) + aZ 



yi(m+di+d2) + aZ, 1 <m < ds + d^ + l, 



where l{k) G (fc + l)m + NZ. We remark that /(O) = m, l{di + d2 + 
ds + d^ + l) = 1{N — 1) = and l{di + ^2) = di + d^ + l, where the last 
equality follows from ( |7.3 ). 



Proof of l^gj] ) and (71^ . The conclusion ( ffiMj ) for A; = from ( [7!29| ) 
and the observation that /(O) = m. Inductively, we assume that the 
conclusion ( 7.31[ ) holds for some < k < di + d2 — 1. Then l{k) 7^ 
0, di + 4 + 1 as l{di + d2) = di + ds + l and 1{N - 1) = 0. If < l{k) < 
di + ds + 1, then 



yi{k+i) = yi(k) + {m + l){b - a + S) + (m - m 

(7.33) = y^k) + ci + b-a 

if l{k + 1) — l{k) = m, and 

yiik+i) + a = yi(k) + im + l){b-a + S) 

(7.34) = y/(fc) + ci + b-a 



1)6 + mh 



(m 



m 



1)6 + mh 



if l{k + 1) — l{k) = m, — N, where (7.33) and (7.34) hold as we have 
inserted m + 1 gaps of size b — a + 6 and m — (m + 1) gaps of size 6 



on [l{k)h, {l{k) + m)h) by Claim 7.2 Also we obtain from (7.30) that 
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yi{k) G [0,co + a — b — 6) when < l{k) < di + ds + 1, which together 
with the inductive hypothesis imphes that 

{Ra,b,c)''^^{c - Co) +aZ = Ra,b,c{yi(k)) + Ct^ 

(7.35) = yii^k) + ci + 6 - a + oL. 
Combining ( 7.33[ ), (7.34) and (7.35) leads to 

(7.36) (i?a,6,c)'+'(c - Co) + aZ = ^((^+1) + aZ 

if < l{k) < rfi + 4 + 1. 

Similarly ii di + d-^ + 1 < l{k) < A^ — 1, we have that 

(7.37) yi{k+i) - yi(k) eci + aZ 

because we have inserted m gaps of size b — a + 6 and m — m gaps of 
size S on [l{k)h, {l{k) + m)h) by Claim 7.2; and 

(7.38) {Ra,b,cf^^{c - Co) + aZ = y^^^) + Ci + aZ, 



since yi{^k) ^ [co,a) by (7.30). Combining (7.37) and (7.38) yields 
(7.39) {RaAcf^\c - Co) + aZ = ^^(^+1) + aZ 

ii di + d^ + 1 < l{k) < N — 1. Therefore we can proceed our inductive 
proof by (7.36) and (7.39). This completes the proof of the equalities 
in (fllh. 



Notice that yi{d-,+d2) = ydi+ds+i = Co + a-b-Shy ( |7.30[ ). Hence 
{Ra,b,cr{co + a-b-6) + aZ= {Ra,b,cr{yi(d^+d,)) + aZ 
1(0)) -\- u^ = yna,b,c) ' ~yc-Co) 



(7.40) = iRa,b,cr^''''-'''iym) + aZ= {Ra,b,cr^'''^''Hc - Co) + aZ 



for all 1 < rra < (is + (i4 + 1 . Then we can follow the argument to prove 
( [73I| to show that ( [732| holds. D 

Finally from (7.31[) and (7.32) the mutually disjoint gaps we have 



inserted are {Ra,b,c 



Co J 



[0, 6 - a + (5) + aZ, < A; < c/i + c?2, and 



(^a,6,c)'^(co + a-b-6) + [0,6) +aZ,l <m <ds + d4 + l. Mo reover 
lRa,b,cY''^'^^{c-Co) + [0,b-a + 6) + aZ= [co + a-&-(5, Co)+aZ by ( |7.30[ ) 
and'/fdi+rfa) = di+d^+l; and iRa,b,cY^^'^*'^^ico+a-b-5) + \0 5)+ aZ = 
{Ra,b,c)^~\co + a-b-6) + [0,6) + aZ = [0,6) + aZ by ^TAO^ and 
1{N — 1) = 0. Notice that the union of the above gaps is invariant 
under the transformation Ra,b,c and contains the black holes of the 
transformations Ra,b,c and Ra,b,c- Therefore its complement is the set 
Sa,b,c by Lemma 5.11, whose restriction on [0, a) has Lebesgue measure 
Nh. Thus the conclusion that Sa,b,c 7^ is established for this case. 

Case 4: There exist nonnegative integers di,d2,d^,d4^ and 6 G 
(- m.iYi{Bd/N, a -Co), 0) satisfying ([7l|-([7^. 
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Set h= {a-{di + d2 + l){b - a))/N + 5 and m = {Nci + (rfi + 4 + 
1)(6 — a)) /a. We expand the real line M with marks at /iZ by inserting 
gaps [6 + a — b,0) located at Imh + NhZ, 1 < I < di + d2 + l, and gaps 
[S, 0) otherwise. After performing the above augmentation operation, 
the interval [0, Nh) with marks [0, Nh) fl/iZ becomes the interval [0, a) 
with gaps [yi + hi,yi),0 < i < Ni, where < yi < . . . < yj\f = a and 
hi E {6 + a — b,6},l < i < N. We follow the argument used in Case 3 
to show that ym = Ci + b — a, yd^^+ds+i = Cq — 6 and for < A; < A^ — 1, 

(7.41) {Ra,b,c)\c - Co + 6) + aZ = yi(^k) + aZ, < k < di + ds, 
and 

(7.42) {RaMrico -6) + aZ = yi^k) + aZ, 1 < m < 4 + c?4 + 1, 

where l{k) G (A; + l)m + A^Z. Therefore Sa^b,c is the complement of 
( Ut+o''' [yi^k) +a-b + 6, yi(k)) +aZ)u{ uttt'^^ [ym + 5, Viik)) + aZ) , 
whose restriction on [0, a) has Lebesgue measure Nh > 0. This prove 
the sufficiency for Case 4. 

Case 5: There exist nonnegative integers ^1,^2,^3,^4 and 6 = 



satisfying (7.1)-(7.6) 



In this case, we set h = {a — {di + d2 + 1)(6 — a))/N and m = 
{N ci + {di + d^ + 1) {b — a)) / a, and expand the real line M with marks at 
hZ by inserting gaps [0, b — a) located at Imh+NhZ, 1 < I < di + d2 + l, 
and gaps of zero length otherwise, c.f. the fourth subfigure of Figure |2] 
in Section [2j Then after performing the above operation, the interval 
[0, Nh) becomes the interval [0, a) with gaps [yi,yi + hi),0 < i < N — 1, 
where = yo < yi < . . . < yN-i and hi G {6 — a, 0}, < i < N—1. We 
follow the argument in Case 3 to show that ym = Ci, ydi+da = Co + a — b 
and by induction onO<k<N — 1 that 

(7.43) {Ra,b,c)''ic - Co) + aZ = yi(k) + aZ, 0<k<di + ^2, 
and 

(7.44) (i?a,fex)™(co) + aZ = yi^m+dr+d^) + ci'^, 1 < m < 4 + 4 + 1, 

where l{k) G (A; + Vjim + A^Z. Thus the union of gaps of size b — a 
is ut^^^\Ra,b,cT{[c -co,c-co + b-a) + aZ with (i?,,b,c)'^+'^([c - 
Co,c — Cq + b — a) + aZ) = [cq + a — b, Cq) + aZ. Therefore Sa^^c is the 
complement of the above union of finite gaps and the sufficiency in the 
fifth case follows. D 
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7.2. Proof of Theorem [2T5| (XIII): By Theorem[3^ G{X[o,c), Q^x 

Z/6) is not a Gabor frame if and only if Va^b,c 7^ 0, which in turn 
becomes Sa,b,c ^ and ( [c/6J + l)\Sa,b,c fl [0, cq + a - 6) | + [c/b\ \Sa,b,c H 



[co,a)| 7^ a by ( 3.27^ and Theorem 5.7, For the case that the triple 
{a,b,c) satisfies the first condition in Theorem |7.1 it follows from the 
argument used in the proof of Theorem 7.1 that Sa,b,c^[0, Co + a — b) = 
and Sa,b,c n [co,a) = U^o [co,gcd(a, ci)) + zgcd(a,ci), where A^ + 1 = 
a/gcd(a,ci). Hence (5.18) holds if and only if (A^ + 1) [c/6J(gcd(a, ci) — 
Co) = a if and only if [c/6J (gcd(a, Ci) — Cq) = gcd(a,ci). For the case 
that the triple {a,b,c) satisfies the second condition in Theorem 7.1 
Sa,b,c n [co, a) = and Sa,b,c n [0, Cq + a - 6) = U,^o[0, gcd(ci + b,a) - 



b + Co) + igcd(ci + b,a), where A^ = a/gcd(ci +b,a) — 1. Hence (5.18) 
holds if and only if {N + l)(Lc/6J + l)(gcd(ci + b,a) - b + cq) = a ii 
and only if ( [c/b\ + l)(gcd(ci + b,a) — b + cq) = gcd(ci + b,a). For the 
case that the triple (a, b, c) satisfies the third condition in Theorem 7.1 
there are di + d^ + l intervals of length h contained in [0, co + b — a) and 
c?2+c?4+l intervals of length h contained in [cq, a), where h+\6\ = B^/N . 
Therefore ( [5l8| ) holds if and only if ([c/6J +I)(rfi + rf3 + l)/i+ Lc/6J(ci2 + 
d^ + l)h = a if and only ii h = a/{\c/b\N + [di + (is + 1)) if and only 
if a/{[c/b\N + {di + c/3 + 1)) + \6\ = Bd/N. Therefore the conclusion 



(XIII) holds by Theorem 7.1 



(XIV): This conclusion is given in [251 Section 3.3.6.1]. 

Appendix A. Algorithm 

In this appendix, we provide a finite-step process to verify whether 
the Gabor system ^(^[o^c),'^^ x Z/&) is a Gabor frame for any given 
triple of (a, 6, c) of positive numbers. 

By Theorems |2 . lH2l5| it suffices to consider triples satisfying assump- 
tions in Conclusion (XII) of Theorem 2.4 and in Conclusion (XIII) of 
Theorem 2.5 For a triple {a,b^c) of positive numbers satisfying as- 
sumptions in Conclusion (XII) of Theorem 2.4, we let Aq = [ci,Ci + 
b — a) (the black hole of the piecewise linear transformation Ra,b,c in 
(2.14)) and 5*0 = [0,a)\y4o, and define holes A^ := [o-k, Pk) C [0, a) and 



invariant sets Sk C [0,a),/c > 1, iteratively 

Ra,b,c{ak^i) - [(-RaAc(«fc-i)/a)Ja +[0,b- a) 

if Ak-i C [0, Co + a - b) or Ak^i C [cq, a), 

Ak-i if Ak-i = [co + a-b, cq), 



A, 



[0,a) 



otherwise, 



and Sk = Sk-i\Ak, k > I. By Theorem 5.4, there exists a nonnegative 
integer L < a/{b — a) such that A^ is invariant for k > L, which implies 
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that Sl is the maximal invariant set Safi,c in (2.9) and Sk = Sl for all 
k > L. Thus by Theorems 3.2 and 5.7, Q{x[o,c),0''^ x Z/6) is a Gabor 
frame if and only if either Sl = ^ or ([c/6J + 1) I^l fl [0,co + a — b)\ + 
[c/6J|S'l n [co,a)\ = a. 

For a triple (a, 6, c) of positive numbers that satisfies assumptions 
in Conclusion (XIII) of Theorem 2.5, we let Bq = [ci, Ci + b — a) and 
To = [0,a)\-Bo, and define holes B^ := [jky^k) C [0, a) and invariant 
sets Tfc C [0,a),k > 1, iteratively 

Ra,b,cilk-l) - L(-Ra,b,c(7fc-l)/a)J«+ [0,4-1 - Ik-l) 

if < 7fc--i < 6k-i < co + a-b, 

Ra,b,c{lk-l) - [{Ra,b,c{lk-l) / a)\ A + [0, Cq + a - 6 - 7fe_i) 

if < 7fe_i < Co + a - 6 < 4-i < cq, 
fife_i if Co + a - 6 < 7fc_i < 4_i < Co, 

c- [c/aja+ [0,4_i -Co) 

if Co + 6 - a < 7fc_i < Co < 4-i < a, 

Ra,b,c{nk-l) - [{.Ra,b,c{nk-l) / a)\a + [0,4_i -7A:-i) 

if Co < 7a:-i < 4-1 < a, 
[0, a) otherwise, 

and Tk = Tk-i\Bk,k > 1. By (5.16) and Theorem 5.5, there exists a 
nonnegative integer L such that Bk C [co + a 



Bh 



b, Co) or Bk = [0, a) 
for all k > L, which implies that Tl is the maximal invariant set 
Sa,b,c ill (2.9). Therefore by Proposition 4.2 and Theorems 3.2 and 
^(X[o,c),oZ X Z/6) is a Gabor frame if and only if either Tl = 

{[c/b] + 1)\Tl n [0, Co + a - 6)1 + [c/b\ \Tl n [cq, a)\ = a. 



5.7 



or 



Appendix B. Sampling signals in a shift-invariant space 

An interesting problem in sampling is to identify generators and 
sampling-shift lattices aZ x 6Z such that any signal / in the shift- 
invariant space 

(B.l) y2(0,6Z):={5^rf(A)0(t-A): J^ M(A)p < oo} 

xebz Ae6Z 

can be stably recovered from its equally-spaced samples /(to + /w), /U ^ 
aZ, for any initial sampling position to, that is, there exist positive 
constants A and B such that 

(B.2) ^ll/l|2<(5^|/(to + ^)r)'^'<5||/||2 

for all / G V2(0, 6Z) and to G M. The range of sampling-shift parame- 
ters a and b is fully known only for few generators 0. For instance, the 
classical Whittaker-Shannon-Kotel'nikov sampling theorem states that 
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(B.2) hold for signals bandlimited to [—a, a] if and only ii a <h = -n /a. 



For the uniform spline generator X[Q,b) * ■ ■ ■ * X[o,6) obtained by convo- 



luting the characteristic function on [0, h) for n times, (B.2) hold if and 
only if a < 6 [U |36]. In this appendix, we consider the above range 
problem for the generator %/, the characteristic function on an interval 
/. Our results indicate that it is almost equivalent to the afec-problem 
for Gabor systems, and hence geometry of the range of sampling-shift 
parameters could be arbitrary complicated. 

We say that {0(- — A) : A G 6Z} is a Riesz basis for the shift-invariant 
space V2{4>,b7j) if there exist positive constants A and B such that 



(B.3) A{ J2 \d{\)\'Y^" < II E dW^^- - A)||2 <B{J2 \dWn 



|^/^<||5:rf(A)0(.-A)||,<5(5:|--'^^^/^ 
AefeZ AebZ AebZ 



for all square-summable sequences {d{X))x^bz- For an interval I = 
[d, c+d), {xi{'~^) '■ ^ ^ b7j} is a Riesz basis for the shift-invariant space 
V2{xh b'^) except that 2 < c/b G Z. Therefore except that 2 < c/b G Z, 
any signal / in V2{xi,b1j) can be stably recovered from its equally- 
spaced samples /(to + l^)il^ £ a^, for any initial sampling position 



to if and only if infinite matrices ^a,b,c{t)it ^ "^i in (2.1) have the 



uniform stability property (3.4), c.f. [21 EH EH]- This together with the 
characterization of frame property of the Gabor system Q{xi, Q-Z x Z/6) 
in [32] leads to the following equivalence between our sampling problem 
associated with the box generator xi and the afcc-problem for Gabor 
systems. 

Proposition B.l. Let a,b > and I be an interval with length c > 0. 
Except that 2 < c/b G Z, the following two statements are equivalent. 

(i) Any signal f in the shift-invariant space V2{xi, &Z) can be stably 
recovered from equally- spaced samples f(to + fi),fj, G aZ, for any 
initial sampling position to G M. 
(ii) Q{xi,a7j x Z/6) is a Gabor frame for L^(]R). 

If / = [d,c + d) with 2 < c/b G Z, then the shift-invariant space 
^2(Xi')^^) is not closed in L^(]R), but its closure is the shift-invariant 
space generated by xi' where /' = [d,d + b). Therefore for the case that 
I = [d,c-'rd) with 2 < c/b G Z, any signal / in V2(x/, bZ) can be stably 
recovered from equally-spaced samples /(to + /i), /i G aZ, for any initial 
sampling position to G M if and only if any signal / in V2{xid,b+d) ^ bZ) 
can be stably recovered from equally-spaced samples /(to + /u), /i G aZ, 
for any initial sampling position to G M if and only if a < 6. This 



together with Theorems 2.1 2.5 and Proposition B.l [provides the full 
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classification of sainpling-sliift lattices oL x \fL such that any signal 
/ in V2(x/,&^) can be stably recovered from equally-spaced samples 
f {to + fJ') , fJ' & ctZ, for any initial sampling position to ^ I^- 



We remark that two statements in Proposition B.l are not equivalent 
for the case that 2 < c/b G Z and a < b. The reason is that under 
that assumption on the triple (a, b, c), Q{xij ^tZ x Z/b) is not a Gabor 



frame by Theorems 2.1 and 2.2, while any signal / in V2{xi,bZi) can 
be stably recovered from equally-spaced samples f (to + fi) , fi G aZ, for 
any initial sampling position to G M. 

Oversampling, i.e., a < b, helps for perfect reconstruction of band- 
limited signals and spline signals from their equally-spaced samples [H 
121 [37] . Our results indicate that oversampling does not always implies 
the stability of sampling and reconstruction process for signals in the 
shift-invariant space V2{xi,bZ). 

Appendix C. Non-ergodicity of a piecewise linear 

transformation 

Recall that the piecewise linear transformation Ra,b,c is non-expansive 
and non-measure-preserving map on the real line, In this appendix, we 
study non-ergodicity associated with this new map on the line. Par- 
ticularly, we establish the equivalence between the empty set property 
for the maximal invariant set Sa,b,c and non-ergodicity of the piecewise 
linear transformation Ra,b,c- The reader may refer to [38] for ergodic 
theory of various dynamic systems. 

Theorem C.l. Let {a,b,c) be a triple of positive numbers satisfy- 
ing a < b < c,b — a < Co '■= c — [c/b\b < a,0 < Ci := [c/b\b — 
[{[c/b\b/a)\a < 2a — b and [c/b\ > 2, and let Ra,b,c and Sa,b,c be the 
piecewise linear transformation in (2.1^ and its maximal invariant set 



m 



(2.^ respectively. Then Sa,b,c = ^ if and only if for any t eM. there 



exists to G [co + a — b, Co) such that 

(C.l) lim ^^k=ofiiRa,b,c)''{t)) ^ ^(^^) 

n— s>oo 11 

for all continuous periodic functions f with period a. 

Proof, (i)^^(ii) Clearly it suffices to prove that for any t G M there 
exists a nonnegative integer A^ such that 

(C.2) {Ra,b,cf{t) G [Co + a - 6, Co) + aZ. 

Suppose, on the contrary, that such a nonnegative integer A^ does not 
exist. Then {Ra,b,c)"' (t) ^ [co + a — b,Co) + aZ for all n > 0. Define 
X = (xt(A))A6ra by Xj(A) = 1 if A = {Ra,b,cT{t)-t for some nonnegative 
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integer n, and Xi(A) = otherwise. Then xj G Bb as {Ra,b,c)"' (t) 
6Z. Following the argument in Proposition 43, we have that 

(C.3) Ma,bA't)Mf^) = 1 for all < /i G aZ. 

Similar to the index Qa,b,c in (3.6), we define 

Qa,b,c ■= sup sup Qa,b,c(i,x) 
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where K{t,:>i.) = {fi E aZ : Ma^b^c(i)x(/i) = 1} and 



Q(t,x) 





supjn G A^l [/i, fi + no) C -/^(t, x) 
for some /i G aZ} 



if kit, x) = 



otherwise. 



Following the argument in Lemma 3.4, we can establish the following 
equivalence: 



(C.4) 



Sa,b,c = if and only if Qa,b,c < oo. 



Therefore combining (C.3 ) and (C.4) leads to the conclusion that Sa,b,c 7^ 
0, which is a contradiction. 

(ii)^^(i) We examine two cases a/h G Q and a/h ^ Q to verify the 
empty-set property for Sa,b,c- 

Case 1: a/h G Q. 

Suppose, on the contrary, that Sa,b,c 7^ 0- Write a/h = p/q for some 
coprime integers p and q, take t G Sa^^c, and define tk = {Ra,b,c)'^ (t) , k > 



0. Then tk ^ [cq + a — h, cq) + aZ for all fc > by Proposition 4.3 



Observe that for any A; > 0, there exists an integer < r^ < p such 
that tk — t — —h G aZ as (tk — t)/h G Z and a = {p/q)h. Therefore 
there exist two integers rii < n2 such that t^ — 1„2 ^ ^Z, which together 
with the one-to-one correspondence of the transformation Ra,b,c on Sa^b,c 
implies that t„2-ni ~ ^o ^ ct^- Thus 



(C.5) 



lim 



ETJofiiRaAcntn.)) ^ET'''''' ''^^ - '" 



^k=0 



fiiRa,b,c)''ito)) 



n 



^2 -rii 



and the limit in (C.l ) does not hold for any continuous periodic function 
/ that it is positive on [cq + a — h — e, cq) and take zero value on 
[0,co + a — h — e) and [cq, a), where e > is sufficiently small. This is 
a contradiction. 

Case 2: a/h ^ Q. 

Take to G M, and let t^o E [cq + a — h, cq) such that 



(C.6) 



lim 

n—>oo 



Er=ofiiRa,b,cnto)) 



n 



f{tc 
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for all continuous periodic function / with period a. We exam three 
subcases to verify that to ^ <Sa,b,c- 

Case 2a: too ^ (cq + a — b, Cq). 

Take a continuous periodic function / such that it is positive on 
(co + a — b, Co) and take zero value on [0, co + a — b) and [co, a). Then 



it follows that ( |C.6[ ) that /((i?a,6,c)''°(to)) > for some h > 0, which 
implies that (-Ra,6,c)'^°(to) belongs to the black hole [co + a — b, cq) + aZ 
of the transformation Ra^b,c- Hence to ^ <Sa,b,c because Sa^^c is invariant 
on the transformation Rafi,c, and Sa^b,c has empty intersection with its 
black hole [cq + a — b, Cq) + oL. 

Case 2b: too = Co + a — 6 and {Ra,b,c)''° (^o) ^ [co + a — 6 + a, cq) + aZ 
for some nonnegative integer fco- 

In this case, to ^ i^a^^j^c as 5a,b,c is invariant on the transformation 
Ra,b,c and [cq + a - b, cq) n Sa,b,c = 0- 

Case 2c: too = Co + a — b and t^ := {Ra,b,c)'' (to) ^ [co + a — b, Cq) +aZ 
for all nonnegative k > 0. 

Take a sufficiently small positive number e > and a periodic func- 
tion /e(t) whose restriction on [0, a) is given by max(0, 1 — |t — co + a — 
6|/e). The function /^ is the hat function supported in [— e, e] + Co + a — 



b + aZ. By (C.6), for any given L there exists an integer k such that 



tk+i G (co + & — a — e, Co + 6 — a) + aZ for all < / < L, as otherwise 

n->-oo n L 

As tfc+/ G (co + 6 — a — e, Co + 6 — a) + aZ for all < / < L, we 
have that t^+i = {Ra,b,cYitk) = 4 + K[c/b\ + 1)6,0 < I < L. Thus 
/([c/6J + 1)6 G (— e,e) + aZ for all < / < L, which is a contradiction 
as L can be chosen sufficiently large and a/b ^ Q. D 
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